Oct 30 17:56:05.443726 ip-10-0-136-241 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:56:05.817160 ip-10-0-136-241 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:56:05.817160 ip-10-0-136-241 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:56:05.817160 ip-10-0-136-241 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:56:05.817160 ip-10-0-136-241 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:56:05.821613 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821547 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.821613 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821613 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.821613 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821617 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821621 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821625 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821630 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821633 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821636 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821639 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821642 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821645 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821647 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821650 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821653 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821655 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821659 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821671 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821674 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821677 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821680 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821683 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.821707 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821686 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821688 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821691 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821694 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821696 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821699 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821702 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821704 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821707 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821709 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821712 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821715 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821719 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821722 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821724 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821727 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821729 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821732 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821736 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821739 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.822214 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821741 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821744 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821747 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821749 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821752 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821754 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821757 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821760 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821762 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821765 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821767 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821770 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821772 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821775 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821778 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821781 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821784 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821786 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821789 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.822909 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821792 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821794 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821797 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821800 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821802 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821805 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821808 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821810 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821812 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821815 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821817 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821820 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821823 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821825 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821828 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821830 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821833 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821835 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821838 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821840 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.823441 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821843 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821845 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821848 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821850 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821852 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821855 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821857 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821861 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821864 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821867 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821870 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821872 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821875 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821878 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821880 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821882 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821885 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821888 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821890 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821893 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.824395 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821895 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821898 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821900 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821902 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821905 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.821908 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824006 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824016 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824020 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824023 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824026 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824028 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824031 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824033 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824036 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824040 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824043 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824045 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824048 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824050 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.824885 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824053 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824056 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824058 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824061 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824063 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824066 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824070 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824074 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824077 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824080 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824083 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824086 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824089 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824091 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824095 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824098 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824101 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824104 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824106 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.825406 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824110 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824112 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824115 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824117 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824120 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824122 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824125 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824127 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824130 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824132 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824135 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824138 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824140 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824143 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824145 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824148 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824150 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824153 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824156 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824158 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.825886 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824160 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824163 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824165 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824168 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824171 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824174 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824176 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824179 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824182 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824185 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824188 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824190 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824193 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824196 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824199 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824202 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824204 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824206 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824209 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824211 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.826389 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824214 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824216 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824219 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824221 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824224 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824226 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824229 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824231 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824234 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824236 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824241 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824246 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824248 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824251 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824254 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824256 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824259 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824261 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824264 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.826878 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824266 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824269 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824271 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824274 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824276 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824280 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824282 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824285 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824288 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824290 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824293 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824295 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824297 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824300 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824806 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824812 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824815 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824818 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824821 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.827407 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824823 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824827 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824842 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824847 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824852 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824857 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824860 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824862 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824865 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824868 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824871 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824874 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824876 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824879 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824882 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824884 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824887 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824890 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824893 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.827881 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824896 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824898 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824901 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824904 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824907 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824909 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824912 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824914 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824917 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824919 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824922 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824925 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824927 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824930 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824932 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824934 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824938 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824942 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824947 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.828346 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824949 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824954 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824957 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824960 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824963 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824966 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824968 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824971 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824974 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824976 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824979 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824981 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824985 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824987 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824990 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824992 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824995 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.824997 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825000 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825002 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.828818 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825005 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825008 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825010 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825013 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825015 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825018 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825020 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825023 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825026 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825028 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825031 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825033 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825036 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825041 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825043 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825046 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825048 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825056 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825059 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.829298 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825061 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825064 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825066 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825069 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825072 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825075 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825077 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825081 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825084 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825086 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825089 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825092 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825094 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825097 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825099 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825102 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825104 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825107 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825109 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825111 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.829766 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825114 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825117 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825119 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825122 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825213 2574 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825221 2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825233 2574 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825239 2574 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825243 2574 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825255 2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825260 2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825264 2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825268 2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825271 2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825274 2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825278 2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825281 2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825284 2574 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825287 2574 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825290 2574 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825293 2574 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825296 2574 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825300 2574 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825303 2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:56:05.830232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825306 2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825309 2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825312 2574 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825316 2574 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825319 2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825322 2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825326 2574 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825329 2574 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825332 2574 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825335 2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825338 2574 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825341 2574 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825345 2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825348 2574 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825353 2574 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825357 2574 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825360 2574 flags.go:64] FLAG: --enable-server="true" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825362 2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825367 2574 flags.go:64] FLAG: --event-burst="100" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825370 2574 flags.go:64] FLAG: --event-qps="50" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825373 2574 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825376 2574 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825379 2574 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825383 2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:56:05.830818 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825386 2574 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825391 2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825395 2574 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825398 2574 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825401 2574 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825404 2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825407 2574 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825410 2574 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825413 2574 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825416 2574 flags.go:64] FLAG: --feature-gates="" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825425 2574 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825428 2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825431 2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825434 2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825438 2574 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825441 2574 flags.go:64] FLAG: --help="false" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825444 2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-241.ec2.internal" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825447 2574 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825450 2574 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825453 2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825469 2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825474 2574 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825478 2574 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:56:05.831431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825481 2574 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825485 2574 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825489 2574 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825492 2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825495 2574 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825498 2574 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825501 2574 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825504 2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825507 2574 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825510 2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825514 2574 flags.go:64] FLAG: --lock-file="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825517 2574 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825520 2574 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825523 2574 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825529 2574 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825532 2574 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825535 2574 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825538 2574 flags.go:64] FLAG: --logging-format="text" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825541 2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825545 2574 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825547 2574 flags.go:64] FLAG: --manifest-url="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825550 2574 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825555 2574 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825558 2574 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825562 2574 flags.go:64] FLAG: --max-pods="110" Oct 30 17:56:05.831999 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825565 2574 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825568 2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825571 2574 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825574 2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825577 2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825579 2574 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825583 2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825591 2574 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825594 2574 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825598 2574 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825601 2574 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825603 2574 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825606 2574 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825609 2574 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825613 2574 flags.go:64] FLAG: --port="10250" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825616 2574 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825618 2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0584a2fed4f43192f" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825623 2574 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825626 2574 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825629 2574 flags.go:64] FLAG: --register-node="true" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825632 2574 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825636 2574 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825638 2574 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825641 2574 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825644 2574 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825648 2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:56:05.832602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825651 2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825654 2574 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825657 2574 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825660 2574 flags.go:64] FLAG: --runonce="false" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825663 2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825666 2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825669 2574 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825672 2574 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825674 2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825677 2574 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825681 2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825684 2574 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825688 2574 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825691 2574 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825694 2574 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825697 2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825701 2574 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825704 2574 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825707 2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825713 2574 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825716 2574 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825718 2574 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825722 2574 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825726 2574 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825729 2574 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:56:05.833228 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825732 2574 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825735 2574 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825738 2574 flags.go:64] FLAG: --v="2" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825743 2574 flags.go:64] FLAG: --version="false" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825747 2574 flags.go:64] FLAG: --vmodule="" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825750 2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.825754 2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825905 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825909 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825913 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825916 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825919 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825921 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825924 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825927 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825930 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825932 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825935 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825937 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825942 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825945 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.833856 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825948 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825950 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825953 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825957 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825961 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825965 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825968 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825971 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825974 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825978 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825981 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825984 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825987 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825989 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825992 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825994 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.825997 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826000 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826002 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.834354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826005 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826008 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826010 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826013 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826015 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826018 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826021 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826023 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826025 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826028 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826032 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826037 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826040 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826042 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826045 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826047 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826050 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826053 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826056 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.834900 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826059 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826061 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826063 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826068 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826070 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826073 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826075 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826078 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826080 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826083 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826085 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826088 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826091 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826093 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826096 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826098 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826101 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826103 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826106 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826108 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.835380 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826111 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826114 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826116 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826118 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826122 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826125 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826128 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826130 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826133 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826136 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826138 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826141 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826144 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826146 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826149 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826153 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826155 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826158 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826160 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826163 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.835872 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826165 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826168 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826170 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826173 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826175 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826178 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826181 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826183 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826186 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826188 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826191 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826193 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826196 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826198 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.826224 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826266 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.836344 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826270 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826274 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826278 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826280 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826283 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826286 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826288 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826291 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826294 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826296 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826299 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826303 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826306 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826308 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826311 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826313 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826316 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826318 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826321 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.836731 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826324 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826327 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826331 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826334 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826337 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826340 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826343 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826346 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826348 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826351 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826354 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826356 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826359 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826361 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826365 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826368 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826371 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826373 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826376 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.837188 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826379 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826381 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826384 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826395 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826398 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826402 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826405 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826407 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826412 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826415 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826418 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826421 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826424 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826427 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826429 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826432 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826435 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826438 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826440 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.837665 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826443 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826445 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826448 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826450 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826453 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826455 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826472 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826475 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826479 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826481 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826484 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826486 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826489 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826491 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826494 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826497 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826499 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826503 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826505 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826508 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.838114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826511 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826514 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826516 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826519 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826521 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826523 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826526 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826529 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826531 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826533 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826536 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826538 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826541 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826544 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826546 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826548 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826551 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826554 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826556 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826558 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.838603 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826561 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826564 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826566 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826568 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826572 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826575 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826578 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.826580 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.832596 2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.832611 2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.832625 2574 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.832630 2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832685 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832690 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832694 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832696 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832700 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832703 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832706 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832708 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.839072 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832711 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832714 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832716 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832719 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832722 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832724 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832727 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832730 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832733 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832735 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832738 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832740 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832743 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832746 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832748 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832751 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832754 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832758 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832761 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.839565 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832764 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832767 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832770 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832773 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832776 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832779 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832781 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832784 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832786 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832789 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832792 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832794 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832797 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832799 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832802 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832805 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832807 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832810 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832812 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.840009 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832816 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832818 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832821 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832824 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832826 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832829 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832831 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832834 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832836 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832839 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832841 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832844 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832848 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832853 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832856 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832860 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832863 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832866 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832868 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832871 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.840502 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832873 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832876 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832879 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832881 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832884 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832886 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832889 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832891 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832894 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832896 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832899 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832902 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832904 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832907 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832909 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832912 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832915 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832917 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832920 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.841003 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832922 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832925 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832928 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832930 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832933 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832935 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832938 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832940 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832943 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832946 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832948 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832951 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832954 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832956 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832959 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832961 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832964 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832966 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832969 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832971 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.841484 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.832974 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.833000 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833152 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833157 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833161 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833164 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833167 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833171 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833175 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833178 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833181 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833183 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833186 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833189 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833191 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833194 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.842004 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833196 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833199 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833202 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833204 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833207 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833209 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833212 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833214 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833217 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833220 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833222 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833225 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833228 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833231 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833234 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833236 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833239 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833242 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833252 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833256 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.842651 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833259 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833261 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833264 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833266 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833269 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833272 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833274 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833277 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833280 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833282 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833286 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833291 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833296 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833301 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833306 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833310 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833314 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833318 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833322 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.843354 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833325 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833327 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833331 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833334 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833337 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833339 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833342 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833344 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833347 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833349 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833352 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833355 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833357 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833360 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833362 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833365 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833367 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833370 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833372 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833375 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.843923 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833377 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833380 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833382 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833385 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833387 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833390 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833393 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833395 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833398 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833400 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833403 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833406 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833408 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833411 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833413 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833416 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833419 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833422 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833425 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.844412 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833427 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833430 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833432 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833435 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833438 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833440 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833442 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833445 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833448 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833450 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833453 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833455 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833468 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833471 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.833496 2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833529 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:56:05.844877 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833533 2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833536 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833539 2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833541 2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833544 2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833549 2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833552 2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833555 2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833557 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833560 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833564 2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833569 2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833572 2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833576 2574 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833579 2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833581 2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833584 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833587 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833589 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:56:05.845264 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833592 2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833594 2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833597 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833600 2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833603 2574 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833605 2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833608 2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833611 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833613 2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833616 2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833618 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833621 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833623 2574 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833629 2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833631 2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833634 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833636 2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833639 2574 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833643 2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833646 2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:56:05.845714 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833648 2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833651 2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833653 2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833656 2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833658 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833661 2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833664 2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833671 2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833674 2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833676 2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833679 2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833682 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833684 2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833687 2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833690 2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833693 2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833695 2574 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833698 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833701 2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:56:05.846186 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833703 2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833706 2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833708 2574 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833711 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833713 2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833716 2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833720 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833722 2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833725 2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833728 2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833730 2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833734 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833737 2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833739 2574 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833742 2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833744 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833747 2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833749 2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833752 2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833756 2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:56:05.846674 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833759 2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833763 2574 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833766 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833768 2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833771 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833773 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833776 2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833779 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833781 2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833784 2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833786 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833789 2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833791 2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833794 2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833797 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833799 2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833801 2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833804 2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833808 2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833810 2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:56:05.847137 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833813 2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833816 2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833818 2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833822 2574 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833824 2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833827 2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:05.833829 2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.833877 2574 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.835883 2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.836882 2574 server.go:1014] "Starting client certificate rotation" Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.836979 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.837020 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.838735 2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:56:05.847633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.840150 2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:56:05.849844 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.849829 2574 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:56:05.853385 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.853370 2574 log.go:25] "Validated CRI v1 image API" Oct 30 17:56:05.854186 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.854161 2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:56:05.857782 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.857761 2574 fs.go:135] Filesystem UUIDs: map[3eb9a2ff-93e1-466f-87b8-1b6c46daa136:/dev/nvme0n1p4 48028ed0-b1aa-4235-9874-b0471ad1d03a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:56:05.857857 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.857780 2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 17:56:05.863087 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.862977 2574 manager.go:217] Machine: {Timestamp:2026-10-30 17:56:05.861872532 +0000 UTC m=+0.308211912 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084602 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0abca374b8aaca9fdc6fb6272d1f SystemUUID:ec2c0abc-a374-b8aa-ca9f-dc6fb6272d1f BootID:97be8be2-c13d-43ec-8759-bbafc2aa8a46 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ff:6a:fd:ed Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ff:6a:fd:ed Speed:0 Mtu:9001} {Name:ovs-system MacAddress:66:38:4b:dc:ef:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:56:05.863087 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.863085 2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:56:05.863186 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.863153 2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:56:05.864204 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.864183 2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:56:05.864332 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.864207 2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:56:05.864887 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.864877 2574 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:56:05.864918 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.864889 2574 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:56:05.864918 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.864914 2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:56:05.864972 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.864927 2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:56:05.865700 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.865691 2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:56:05.868633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.868622 2574 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:56:05.868754 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.868745 2574 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:56:05.869316 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.869305 2574 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:56:05.869345 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.869322 2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:56:05.869345 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.869339 2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:56:05.869399 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.869352 2574 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:56:05.869399 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.869364 2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:56:05.869754 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.869736 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:56:05.871746 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.871728 2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:56:05.876213 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.876200 2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:56:05.877013 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877000 2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:56:05.877048 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877031 2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:56:05.877183 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877174 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:56:05.877213 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877189 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:56:05.877213 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877195 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:56:05.877213 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877201 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:56:05.877213 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877206 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:56:05.877213 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877212 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:56:05.877887 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877877 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:56:05.877921 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877889 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:56:05.877921 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877896 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:56:05.877921 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877902 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:56:05.877991 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877926 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:56:05.877991 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.877935 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:56:05.878503 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.878492 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:56:05.878541 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.878507 2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:56:05.882982 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.882969 2574 server.go:1263] "Started kubelet" Oct 30 17:56:05.886769 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.883021 2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:56:05.887220 ip-10-0-136-241 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:56:05.888195 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.887204 2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:56:05.888195 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.887290 2574 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:56:05.890544 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.890215 2574 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:56:05.892140 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.892124 2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:56:05.893425 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.893403 2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 17:56:05.896496 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.896484 2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:56:05.896578 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.896556 2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:56:05.896708 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.896689 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:05.897117 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897096 2574 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:56:05.897117 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897119 2574 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:56:05.897261 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897227 2574 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:56:05.897261 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897252 2574 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:56:05.897403 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897283 2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:56:05.897481 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897408 2574 factory.go:55] Registering systemd factory Oct 30 17:56:05.897481 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897424 2574 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:56:05.897662 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897637 2574 factory.go:153] Registering CRI-O factory Oct 30 17:56:05.897662 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897654 2574 factory.go:223] Registration of the crio container factory successfully Oct 30 17:56:05.897762 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897702 2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:56:05.897762 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897715 2574 factory.go:103] Registering Raw factory Oct 30 17:56:05.897762 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.897725 2574 manager.go:1199] Started watching for new ooms in manager Oct 30 17:56:05.898659 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.898640 2574 manager.go:319] Starting recovery of all containers Oct 30 17:56:05.908091 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.908074 2574 manager.go:324] Recovery completed Oct 30 17:56:05.912912 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.912900 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:05.914436 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914422 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:05.914512 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914446 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:05.914512 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914455 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:05.914978 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914965 2574 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:56:05.914978 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914977 2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:56:05.915085 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914979 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:05.915085 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.914992 2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:56:05.915903 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.915885 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:05.915993 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.915910 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:05.915993 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.915926 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:05.916324 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.916301 2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 17:56:05.916324 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.916315 2574 policy_none.go:50] "Start" Oct 30 17:56:05.916433 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.916340 2574 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:56:05.916433 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.916369 2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:56:05.916690 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.916303 2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 17:56:05.917577 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.916568 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-241.ec2.internal.18e3604e43b062a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-241.ec2.internal,},FirstTimestamp:2026-10-30 17:56:05.882938024 +0000 UTC m=+0.329277403,LastTimestamp:2026-10-30 17:56:05.882938024 +0000 UTC m=+0.329277403,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-241.ec2.internal,}" Oct 30 17:56:05.918147 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.918130 2574 policy_none.go:44] "Start" Oct 30 17:56:05.932882 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.932798 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-241.ec2.internal.18e3604e4590fc3b default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-241.ec2.internal,},FirstTimestamp:2026-10-30 17:56:05.914434619 +0000 UTC m=+0.360774000,LastTimestamp:2026-10-30 17:56:05.914434619 +0000 UTC m=+0.360774000,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-241.ec2.internal,}" Oct 30 17:56:05.933276 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.933236 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 17:56:05.933276 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.933242 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 17:56:05.943738 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.943722 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:56:05.943799 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.943775 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fzr24" Oct 30 17:56:05.945747 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.945726 2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 17:56:05.946359 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.946335 2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:56:05.948804 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.948725 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-241.ec2.internal.18e3604e4591392a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-136-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-136-241.ec2.internal,},FirstTimestamp:2026-10-30 17:56:05.914450218 +0000 UTC m=+0.360789598,LastTimestamp:2026-10-30 17:56:05.914450218 +0000 UTC m=+0.360789598,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-241.ec2.internal,}" Oct 30 17:56:05.949642 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.949628 2574 manager.go:341] "Starting Device Plugin manager" Oct 30 17:56:05.950145 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.950130 2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:56:05.950145 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.950146 2574 server.go:87] "Starting device plugin registration server" Oct 30 17:56:05.950327 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.950317 2574 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:56:05.950360 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.950328 2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:56:05.950694 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.950430 2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:56:05.950694 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.950532 2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:56:05.950694 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.950540 2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:56:05.950956 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.950939 2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:56:05.951041 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.950985 2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:05.953817 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:05.953799 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fzr24" Oct 30 17:56:05.967475 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:05.967317 2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-241.ec2.internal.18e3604e45919adc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-241.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-136-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-136-241.ec2.internal,},FirstTimestamp:2026-10-30 17:56:05.914475228 +0000 UTC m=+0.360814608,LastTimestamp:2026-10-30 17:56:05.914475228 +0000 UTC m=+0.360814608,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-241.ec2.internal,}" Oct 30 17:56:06.032034 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.032010 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:56:06.033173 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.033159 2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:56:06.033232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.033181 2574 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:56:06.033232 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.033198 2574 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:56:06.033347 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.033233 2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 17:56:06.050448 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.050431 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:06.050680 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.050666 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:06.051155 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.051142 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:06.051226 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.051165 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:06.051226 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.051175 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:06.051226 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.051194 2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.073387 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.073342 2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.073387 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.073363 2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-241.ec2.internal\": node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.127141 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.127119 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.134196 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.134176 2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal"] Oct 30 17:56:06.134251 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.134225 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:06.134987 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.134972 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:06.135043 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.134999 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:06.135043 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.135011 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:06.136284 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136271 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:06.136439 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136414 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.136511 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136449 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:06.136964 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136947 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:06.137037 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136972 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:06.137037 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136987 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:06.137037 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.136994 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:06.137037 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.137011 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:06.137037 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.137024 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:06.138015 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.137999 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.138073 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.138025 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:06.138667 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.138651 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:06.138729 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.138677 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:06.138729 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.138691 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:06.159006 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.158984 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-241.ec2.internal\" not found" node="ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.163411 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.163395 2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-241.ec2.internal\" not found" node="ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.198589 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.198572 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2a11081781848cec6e4c7eceee10f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" (UID: \"ef2a11081781848cec6e4c7eceee10f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.198667 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.198593 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.198667 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.198610 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.227531 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.227515 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.299576 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.299552 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.299576 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.299572 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.299727 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.299591 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.299727 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.299614 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2a11081781848cec6e4c7eceee10f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" (UID: \"ef2a11081781848cec6e4c7eceee10f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.299727 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.299645 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2a11081781848cec6e4c7eceee10f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" (UID: \"ef2a11081781848cec6e4c7eceee10f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.299727 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.299643 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.328410 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.328367 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.428617 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.428590 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.462418 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.462399 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.467044 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.467029 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:06.529276 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.529255 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.629745 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.629696 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.730252 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.730233 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.830804 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.830782 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.836961 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.836941 2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 17:56:06.837106 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.837086 2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 17:56:06.931609 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:06.931556 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:06.947126 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.947104 2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:56:06.948230 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.948209 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:56:06.948316 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.948250 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:56:06.948316 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.948266 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:56:06.957309 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.957281 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:51:05 +0000 UTC" deadline="2028-05-31 10:57:24.437995073 +0000 UTC" Oct 30 17:56:06.957309 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.957306 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13889h1m17.480692417s" Oct 30 17:56:06.975107 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:06.975080 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef2a11081781848cec6e4c7eceee10f4.slice/crio-9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851 WatchSource:0}: Error finding container 9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851: Status 404 returned error can't find the container with id 9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851 Oct 30 17:56:06.975531 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:06.975516 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod181ed97c8a1c8f7ef8e42a31d315dd0f.slice/crio-51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312 WatchSource:0}: Error finding container 51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312: Status 404 returned error can't find the container with id 51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312 Oct 30 17:56:06.981183 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:06.981170 2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:56:07.031651 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.031634 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:07.036812 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.036771 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" event={"ID":"181ed97c8a1c8f7ef8e42a31d315dd0f","Type":"ContainerStarted","Data":"51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312"} Oct 30 17:56:07.037720 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.037703 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" event={"ID":"ef2a11081781848cec6e4c7eceee10f4","Type":"ContainerStarted","Data":"9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851"} Oct 30 17:56:07.072451 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.072435 2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:07.131697 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.131677 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:07.232074 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.232032 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:07.332436 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.332417 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:07.433326 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.433301 2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:56:07.434431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.434410 2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:07.496856 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.496801 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:56:07.504822 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.504805 2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:07.517383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.517362 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:56:07.518361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.518344 2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:56:07.538811 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.538793 2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:56:07.682166 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.682142 2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:56:07.870804 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.870777 2574 apiserver.go:52] "Watching apiserver" Oct 30 17:56:07.897337 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.897311 2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:56:07.897722 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.897697 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-bzkm6","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58","openshift-cluster-node-tuning-operator/tuned-jhj27","openshift-dns/node-resolver-l2ddq","openshift-image-registry/node-ca-rd5c5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal","openshift-multus/multus-additional-cni-plugins-cpntx","openshift-network-operator/iptables-alerter-7llnl","kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal","openshift-network-diagnostics/network-check-target-pjfgc","openshift-multus/multus-tg6rw","openshift-multus/network-metrics-daemon-c8fkb","openshift-ovn-kubernetes/ovnkube-node-brcs6","kube-system/global-pull-secret-syncer-gqtbs"] Oct 30 17:56:07.898182 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.898163 2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 17:56:07.900250 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.900229 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.902439 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.902127 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tg6rw" Oct 30 17:56:07.904119 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.904101 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:07.906399 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.906174 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:07.906908 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.906892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.906992 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.906920 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.906992 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.906943 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.906992 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.906967 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6f5xs\" (UniqueName: \"kubernetes.io/projected/64fed540-f4cb-4e19-b3d9-7531e486c4f5-kube-api-access-6f5xs\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.907134 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.906992 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.907134 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.907015 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.907134 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.907048 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cnibin\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:07.908265 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.908187 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:07.910182 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.910164 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:07.912192 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.912173 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:07.912275 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.912250 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:07.914399 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.914382 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.917390 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.917504 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.917746 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.918032 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.918065 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.918442 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26whl\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.918653 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.919110 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pc6fd\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.919156 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.919290 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:07.919361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.919351 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:07.920281 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.920260 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:56:07.920429 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.920416 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:56:07.920688 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.920675 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxlb9\"" Oct 30 17:56:07.921555 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.921535 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:07.921629 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.921614 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:07.922072 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.922056 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vkdz6\"" Oct 30 17:56:07.922154 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.922113 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:56:07.922209 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.922186 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:56:07.922252 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.922241 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:56:07.922693 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.922675 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vdz2\"" Oct 30 17:56:07.924354 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.924248 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:56:07.924354 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.924306 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v7j5r\"" Oct 30 17:56:07.924354 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.924324 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:56:07.924574 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.924401 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:56:07.924574 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.924560 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:07.926511 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.926495 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:07.926601 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:07.926541 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:07.946524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.946507 2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 17:56:07.946610 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.946563 2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-sf5lw" Oct 30 17:56:07.957790 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.957771 2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-sf5lw" Oct 30 17:56:07.998486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:07.998402 2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:56:08.007802 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.007781 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-bin\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.007909 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.007814 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-modprobe-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.007909 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.007835 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysconfig\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.007909 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.007856 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-cni-binary-copy\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.008034 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.007909 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-multus\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.008034 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.007959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-kubelet\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.008034 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008004 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.008034 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008030 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-var-lib-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008211 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008052 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-node-log\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008211 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008091 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-log-socket\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008211 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008104 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-kubernetes\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.008211 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008120 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-netns\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.008211 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008141 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-conf-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.008211 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008196 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfcad2e0-540b-4e42-ba77-303e060d37bd-agent-certs\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.008497 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfcad2e0-540b-4e42-ba77-303e060d37bd-konnectivity-ca\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.008497 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008288 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8tjv\" (UniqueName: \"kubernetes.io/projected/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kube-api-access-z8tjv\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.008497 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008357 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-etc-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008497 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008392 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008497 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008429 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-kubelet-config\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.008497 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008480 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cc87edf-5705-49bc-abe7-7cf921ed75ab-host\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008504 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008529 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-systemd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008542 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlpnz\" (UniqueName: \"kubernetes.io/projected/9d111489-3f0f-49bb-9de2-f400960f01d7-kube-api-access-xlpnz\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008556 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-run\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008570 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-tuned\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008601 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-tmp\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008617 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pm8\" (UniqueName: \"kubernetes.io/projected/9cc87edf-5705-49bc-abe7-7cf921ed75ab-kube-api-access-h6pm8\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008641 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-systemd\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008664 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbst6\" (UniqueName: \"kubernetes.io/projected/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-kube-api-access-tbst6\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008685 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008704 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.008763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5xs\" (UniqueName: \"kubernetes.io/projected/64fed540-f4cb-4e19-b3d9-7531e486c4f5-kube-api-access-6f5xs\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-multus-certs\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008816 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-etc-kubernetes\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91437e-1659-46b1-bc95-687ddd06c392-host-slash\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008852 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008871 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-kubelet\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008902 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-host\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008939 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-socket-dir-parent\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cc87edf-5705-49bc-abe7-7cf921ed75ab-serviceca\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.008977 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-config\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009001 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009032 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-hosts-file\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009049 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-k8s-cni-cncf-io\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-env-overrides\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009102 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91437e-1659-46b1-bc95-687ddd06c392-iptables-alerter-script\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.009264 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009149 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009171 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009217 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009236 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009252 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-script-lib\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009277 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-dbus\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009294 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009317 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-hostroot\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009332 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009347 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxbpv\" (UniqueName: \"kubernetes.io/projected/f4303592-c23b-49bc-a242-dfefafe43d0a-kube-api-access-kxbpv\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009373 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-sys\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009387 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-lib-modules\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009409 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009401 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jjx\" (UniqueName: \"kubernetes.io/projected/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-kube-api-access-s9jjx\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009434 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-tmp-dir\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009498 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009528 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-system-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.009954 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009550 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009590 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009691 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009549 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-cnibin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-netns\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009817 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-netd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9s6l\" (UniqueName: \"kubernetes.io/projected/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-kube-api-access-l9s6l\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009890 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009914 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-daemon-config\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009928 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-ovn\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009942 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-conf\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.009994 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010014 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-bin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010033 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d111489-3f0f-49bb-9de2-f400960f01d7-ovn-node-metrics-cert\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010055 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-var-lib-kubelet\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.010807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010081 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-device-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010099 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cnibin\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgg2b\" (UniqueName: \"kubernetes.io/projected/5b91437e-1659-46b1-bc95-687ddd06c392-kube-api-access-qgg2b\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010121 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010129 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-systemd-units\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010144 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-slash\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010205 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010289 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cnibin\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010425 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mmrm7\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010435 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010514 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010539 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010578 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010606 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z6vc6\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010633 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010696 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9xqn\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.010791 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:56:08.011549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.011164 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.029714 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.029692 2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:56:08.033974 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.033953 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5xs\" (UniqueName: \"kubernetes.io/projected/64fed540-f4cb-4e19-b3d9-7531e486c4f5-kube-api-access-6f5xs\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.110372 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110348 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-k8s-cni-cncf-io\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110384 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-env-overrides\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110405 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91437e-1659-46b1-bc95-687ddd06c392-iptables-alerter-script\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110420 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110432 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110451 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110476 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-k8s-cni-cncf-io\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110493 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110488 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-script-lib\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110508 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-dbus\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110523 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-hostroot\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110536 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbpv\" (UniqueName: \"kubernetes.io/projected/f4303592-c23b-49bc-a242-dfefafe43d0a-kube-api-access-kxbpv\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110549 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-sys\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110561 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-lib-modules\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110572 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jjx\" (UniqueName: \"kubernetes.io/projected/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-kube-api-access-s9jjx\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110584 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-tmp-dir\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-system-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110609 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-cnibin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110620 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-netns\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110632 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-netd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110647 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110659 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9s6l\" (UniqueName: \"kubernetes.io/projected/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-kube-api-access-l9s6l\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110674 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-daemon-config\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110703 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-ovn\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110714 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-conf\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.110751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110730 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-bin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110742 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d111489-3f0f-49bb-9de2-f400960f01d7-ovn-node-metrics-cert\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110754 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-var-lib-kubelet\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110767 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-device-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110781 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgg2b\" (UniqueName: \"kubernetes.io/projected/5b91437e-1659-46b1-bc95-687ddd06c392-kube-api-access-qgg2b\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110856 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-cnibin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110899 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-systemd-units\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110910 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-env-overrides\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110929 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-slash\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110954 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-bin\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110963 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91437e-1659-46b1-bc95-687ddd06c392-iptables-alerter-script\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.110977 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-modprobe-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111000 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysconfig\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111024 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-cni-binary-copy\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111033 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-conf\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111047 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-multus\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111068 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-var-lib-kubelet\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111070 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-kubelet\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111117 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-var-lib-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111139 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-node-log\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111159 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-netns\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111161 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-log-socket\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111182 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-netd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111182 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-kubernetes\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-netns\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-conf-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111232 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfcad2e0-540b-4e42-ba77-303e060d37bd-agent-certs\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111234 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-kubernetes\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111245 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfcad2e0-540b-4e42-ba77-303e060d37bd-konnectivity-ca\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111287 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111314 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tjv\" (UniqueName: \"kubernetes.io/projected/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kube-api-access-z8tjv\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111333 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-conf-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111342 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-slash\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.111685 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111337 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-etc-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111363 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-etc-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111396 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-bin\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111400 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-daemon-config\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111394 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111447 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-ovn\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111471 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111478 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-netns\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111096 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-device-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111494 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-kubelet-config\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111505 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111511 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-modprobe-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111518 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cc87edf-5705-49bc-abe7-7cf921ed75ab-host\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111541 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111545 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-kubelet\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111557 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-node-log\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111563 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-systemd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.111142 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:08.112175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111588 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpnz\" (UniqueName: \"kubernetes.io/projected/9d111489-3f0f-49bb-9de2-f400960f01d7-kube-api-access-xlpnz\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111601 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111610 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-run\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.111627 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:08.61160873 +0000 UTC m=+3.057948112 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111663 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-log-socket\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111706 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-run\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111731 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfcad2e0-540b-4e42-ba77-303e060d37bd-konnectivity-ca\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111880 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111943 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-kubelet-config\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111956 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-script-lib\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111986 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysconfig\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.111998 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-cni-binary-copy\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112031 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cc87edf-5705-49bc-abe7-7cf921ed75ab-host\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112033 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-multus\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112068 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-bin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112066 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-tuned\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112088 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-dbus\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.112981 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.112103 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112105 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-tmp\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112128 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-hostroot\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.112141 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:56:08.612128824 +0000 UTC m=+3.058468202 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112159 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-var-lib-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112173 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-systemd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112311 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-sys\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112347 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-system-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112367 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-systemd-units\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pm8\" (UniqueName: \"kubernetes.io/projected/9cc87edf-5705-49bc-abe7-7cf921ed75ab-kube-api-access-h6pm8\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112397 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-systemd\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112411 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbst6\" (UniqueName: \"kubernetes.io/projected/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-kube-api-access-tbst6\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112468 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112490 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-multus-certs\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-etc-kubernetes\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112532 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91437e-1659-46b1-bc95-687ddd06c392-host-slash\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.113783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112553 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112557 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-tmp-dir\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112575 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-kubelet\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112597 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112616 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-host\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112634 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-socket-dir-parent\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112657 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cc87edf-5705-49bc-abe7-7cf921ed75ab-serviceca\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112679 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-config\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112685 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-lib-modules\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112697 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112718 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-hosts-file\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112788 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-hosts-file\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112831 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-systemd\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.112946 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-kubelet\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113004 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113031 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113050 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-host\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113095 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-socket-dir-parent\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.114341 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113125 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113173 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-multus-certs\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113220 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-etc-kubernetes\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113263 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91437e-1659-46b1-bc95-687ddd06c392-host-slash\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113315 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113470 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cc87edf-5705-49bc-abe7-7cf921ed75ab-serviceca\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113496 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d111489-3f0f-49bb-9de2-f400960f01d7-ovn-node-metrics-cert\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113592 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.113773 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-config\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.114119 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-tuned\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.114126 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfcad2e0-540b-4e42-ba77-303e060d37bd-agent-certs\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.114988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.114536 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-tmp\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.132702 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.132644 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:08.132702 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.132667 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:08.132702 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.132679 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:08.133028 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.132726 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:08.632711907 +0000 UTC m=+3.079051284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:08.133028 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.132924 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tjv\" (UniqueName: \"kubernetes.io/projected/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kube-api-access-z8tjv\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.133028 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.132937 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgg2b\" (UniqueName: \"kubernetes.io/projected/5b91437e-1659-46b1-bc95-687ddd06c392-kube-api-access-qgg2b\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.134004 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.133985 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbpv\" (UniqueName: \"kubernetes.io/projected/f4303592-c23b-49bc-a242-dfefafe43d0a-kube-api-access-kxbpv\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.137388 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.137364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9s6l\" (UniqueName: \"kubernetes.io/projected/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-kube-api-access-l9s6l\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:08.137916 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.137879 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pm8\" (UniqueName: \"kubernetes.io/projected/9cc87edf-5705-49bc-abe7-7cf921ed75ab-kube-api-access-h6pm8\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.138620 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.138603 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbst6\" (UniqueName: \"kubernetes.io/projected/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-kube-api-access-tbst6\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.139383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.139338 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpnz\" (UniqueName: \"kubernetes.io/projected/9d111489-3f0f-49bb-9de2-f400960f01d7-kube-api-access-xlpnz\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.139526 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.139508 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jjx\" (UniqueName: \"kubernetes.io/projected/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-kube-api-access-s9jjx\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.228009 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.227985 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tg6rw" Oct 30 17:56:08.230060 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.230035 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:56:08.233293 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.233273 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:56:08.236332 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.236315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:56:08.238688 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.238665 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4303592_c23b_49bc_a242_dfefafe43d0a.slice/crio-0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33 WatchSource:0}: Error finding container 0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33: Status 404 returned error can't find the container with id 0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33 Oct 30 17:56:08.241782 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.241764 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64fed540_f4cb_4e19_b3d9_7531e486c4f5.slice/crio-6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5 WatchSource:0}: Error finding container 6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5: Status 404 returned error can't find the container with id 6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5 Oct 30 17:56:08.242866 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.242852 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:56:08.244825 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.244804 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d1faa3_7a92_46c7_a3b1_3b1f4e1f8790.slice/crio-c83392f24e59a43564295783a2fbf6adb71cecfc99ab0bf90c51da6cc77d70e6 WatchSource:0}: Error finding container c83392f24e59a43564295783a2fbf6adb71cecfc99ab0bf90c51da6cc77d70e6: Status 404 returned error can't find the container with id c83392f24e59a43564295783a2fbf6adb71cecfc99ab0bf90c51da6cc77d70e6 Oct 30 17:56:08.249840 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.249819 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd7a723_acf0_4c7c_95a5_2bd54d65460d.slice/crio-80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c WatchSource:0}: Error finding container 80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c: Status 404 returned error can't find the container with id 80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c Oct 30 17:56:08.250982 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.250966 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:56:08.253216 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.253197 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cc87edf_5705_49bc_abe7_7cf921ed75ab.slice/crio-0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2 WatchSource:0}: Error finding container 0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2: Status 404 returned error can't find the container with id 0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2 Oct 30 17:56:08.257969 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.257949 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:08.260505 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.260485 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b91437e_1659_46b1_bc95_687ddd06c392.slice/crio-ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932 WatchSource:0}: Error finding container ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932: Status 404 returned error can't find the container with id ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932 Oct 30 17:56:08.268005 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.267989 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:56:08.268690 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.268661 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcad2e0_540b_4e42_ba77_303e060d37bd.slice/crio-6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f WatchSource:0}: Error finding container 6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f: Status 404 returned error can't find the container with id 6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f Oct 30 17:56:08.281114 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.281092 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf0ad50_df01_43e0_bb53_7c51bbcef6b4.slice/crio-e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5 WatchSource:0}: Error finding container e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5: Status 404 returned error can't find the container with id e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5 Oct 30 17:56:08.283026 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.282987 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:08.290128 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:08.290084 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d111489_3f0f_49bb_9de2_f400960f01d7.slice/crio-e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c WatchSource:0}: Error finding container e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c: Status 404 returned error can't find the container with id e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c Oct 30 17:56:08.616487 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.615995 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:08.616487 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.616247 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:08.616487 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.616284 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:08.616487 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.616314 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:09.61629632 +0000 UTC m=+4.062635705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:08.616487 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.616367 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:08.616487 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.616436 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:56:09.616421642 +0000 UTC m=+4.062761023 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:08.717806 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.717234 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:08.717806 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.717413 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:08.717806 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.717439 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:08.717806 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.717450 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:08.717806 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:08.717512 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:09.717496892 +0000 UTC m=+4.163836271 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:08.960726 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.960589 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:51:07 +0000 UTC" deadline="2028-06-02 17:04:15.845844069 +0000 UTC" Oct 30 17:56:08.960726 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:08.960646 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13943h8m6.885201676s" Oct 30 17:56:09.053178 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.053134 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7llnl" event={"ID":"5b91437e-1659-46b1-bc95-687ddd06c392","Type":"ContainerStarted","Data":"ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932"} Oct 30 17:56:09.059022 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.058988 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rd5c5" event={"ID":"9cc87edf-5705-49bc-abe7-7cf921ed75ab","Type":"ContainerStarted","Data":"0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2"} Oct 30 17:56:09.074650 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.074622 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l2ddq" event={"ID":"bbd7a723-acf0-4c7c-95a5-2bd54d65460d","Type":"ContainerStarted","Data":"80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c"} Oct 30 17:56:09.091181 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.091114 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerStarted","Data":"6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5"} Oct 30 17:56:09.109166 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.108928 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tg6rw" event={"ID":"f4303592-c23b-49bc-a242-dfefafe43d0a","Type":"ContainerStarted","Data":"0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33"} Oct 30 17:56:09.120398 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.120370 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" event={"ID":"ef2a11081781848cec6e4c7eceee10f4","Type":"ContainerStarted","Data":"aab41c539eec2087ae934b6ad28636b4d398c9c9ba8c316b1d20055b88e2d6b4"} Oct 30 17:56:09.127807 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.125919 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c"} Oct 30 17:56:09.129623 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.128365 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" event={"ID":"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4","Type":"ContainerStarted","Data":"e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5"} Oct 30 17:56:09.144406 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.144376 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhj27" event={"ID":"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790","Type":"ContainerStarted","Data":"c83392f24e59a43564295783a2fbf6adb71cecfc99ab0bf90c51da6cc77d70e6"} Oct 30 17:56:09.151650 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.151625 2574 generic.go:356] "Generic (PLEG): container finished" podID="181ed97c8a1c8f7ef8e42a31d315dd0f" containerID="68d636785d8954011199b11568c848daf74900e9051dac0eefefa0827f70699b" exitCode=0 Oct 30 17:56:09.151737 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.151702 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" event={"ID":"181ed97c8a1c8f7ef8e42a31d315dd0f","Type":"ContainerDied","Data":"68d636785d8954011199b11568c848daf74900e9051dac0eefefa0827f70699b"} Oct 30 17:56:09.170261 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.170230 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bzkm6" event={"ID":"cfcad2e0-540b-4e42-ba77-303e060d37bd","Type":"ContainerStarted","Data":"6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f"} Oct 30 17:56:09.204076 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.204019 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" podStartSLOduration=2.204002563 podStartE2EDuration="2.204002563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:09.1638934 +0000 UTC m=+3.610232790" watchObservedRunningTime="2026-10-30 17:56:09.204002563 +0000 UTC m=+3.650341953" Oct 30 17:56:09.625339 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.624479 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:09.625339 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.624554 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:09.625339 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.624685 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:09.625339 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.624746 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:11.624728567 +0000 UTC m=+6.071067940 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:09.625339 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.625123 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:09.625339 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.625175 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:56:11.625159476 +0000 UTC m=+6.071498845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:09.725234 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.725202 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:09.725415 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.725337 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:09.725415 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.725356 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:09.725415 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.725368 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:09.725415 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:09.725415 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:11.725399539 +0000 UTC m=+6.171738920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:09.961437 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.961326 2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:51:07 +0000 UTC" deadline="2028-07-08 14:34:30.324948942 +0000 UTC" Oct 30 17:56:09.961437 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:09.961363 2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14804h38m20.363590295s" Oct 30 17:56:10.034195 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:10.034166 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:10.034325 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:10.034291 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:10.034879 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:10.034856 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:10.035017 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:10.034994 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:10.035085 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:10.035064 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:10.035151 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:10.035133 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:10.184069 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:10.184035 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" event={"ID":"181ed97c8a1c8f7ef8e42a31d315dd0f","Type":"ContainerStarted","Data":"a98d29d884af12a100a367a8cfa55856384f7fbb821a63efe82a45a50749a14b"} Oct 30 17:56:11.195358 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:11.195322 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerDied","Data":"d844be6efa2643ea424fe8e728dc969cdbecb922abd4e80fbaf82ebbc9f068af"} Oct 30 17:56:11.195719 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:11.195488 2574 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="d844be6efa2643ea424fe8e728dc969cdbecb922abd4e80fbaf82ebbc9f068af" exitCode=0 Oct 30 17:56:11.269090 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:11.268775 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" podStartSLOduration=4.268756594 podStartE2EDuration="4.268756594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:10.225384314 +0000 UTC m=+4.671723705" watchObservedRunningTime="2026-10-30 17:56:11.268756594 +0000 UTC m=+5.715096001" Oct 30 17:56:11.641994 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:11.641900 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:11.641994 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:11.641979 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:11.642243 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.642096 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:11.642243 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.642155 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.642137467 +0000 UTC m=+10.088476849 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:11.642605 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.642583 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:11.642722 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.642645 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.642629396 +0000 UTC m=+10.088968766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:11.742565 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:11.742528 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:11.742802 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.742785 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:11.742879 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.742809 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:11.742879 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.742822 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:11.743088 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:11.742882 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:15.742858994 +0000 UTC m=+10.189198367 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:12.034693 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:12.034620 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:12.034792 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:12.034742 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:12.035346 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:12.035122 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:12.035346 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:12.035231 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:12.035446 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:12.035371 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:12.035493 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:12.035452 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:12.217393 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:12.217350 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-bzkm6" event={"ID":"cfcad2e0-540b-4e42-ba77-303e060d37bd","Type":"ContainerStarted","Data":"f430dcafa17f78897b2fe4d5c4b805869dda07f1c235d95beb9b1c728af79c12"} Oct 30 17:56:12.258681 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:12.258626 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-bzkm6" podStartSLOduration=3.66343664 podStartE2EDuration="7.258608693s" totalImagesPullingTime="3.595172053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:12.258253258 +0000 UTC m=+6.704592648" watchObservedRunningTime="2026-10-30 17:56:12.258608693 +0000 UTC m=+6.704948083" Oct 30 17:56:13.224252 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:13.224098 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rd5c5" event={"ID":"9cc87edf-5705-49bc-abe7-7cf921ed75ab","Type":"ContainerStarted","Data":"94e134e9a40562d0d1885e071aac6061f2b925d6c15c9a00c5fd4b317cc82a26"} Oct 30 17:56:13.456105 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:13.455949 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:13.456688 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:13.456582 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:13.497606 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:13.497510 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rd5c5" podStartSLOduration=3.870352519 podStartE2EDuration="8.497491445s" totalImagesPullingTime="4.627138926s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:13.26485228 +0000 UTC m=+7.711191671" watchObservedRunningTime="2026-10-30 17:56:13.497491445 +0000 UTC m=+7.943830837" Oct 30 17:56:14.037057 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:14.036555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:14.037057 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:14.036593 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:14.037057 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:14.036555 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:14.037057 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:14.036690 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:14.037057 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:14.036741 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:14.037057 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:14.036807 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:14.229569 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:14.229533 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" event={"ID":"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4","Type":"ContainerStarted","Data":"ba0cbd8415e2e314b13074d3adca6c2d5a7f1c19dbee526650292bbed8575c7d"} Oct 30 17:56:15.233950 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.233303 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:15.417611 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.417583 2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:56:15.676234 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.676189 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:15.676354 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.676272 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:15.676354 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.676303 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:15.676499 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.676357 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.676340063 +0000 UTC m=+18.122679443 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:15.676499 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.676381 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:15.676499 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.676417 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.676407396 +0000 UTC m=+18.122746776 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:15.776642 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.776615 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:15.776804 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.776786 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:15.776871 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.776812 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:15.776871 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.776824 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:15.776972 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:15.776877 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:23.776858502 +0000 UTC m=+18.223197883 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:15.958481 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.958364 2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:56:15.417603519Z","UUID":"602b533a-a1f2-4ace-a158-d467a034c9c6","Handler":null,"Name":"","Endpoint":""} Oct 30 17:56:15.960052 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.960029 2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:56:15.960186 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.960172 2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:56:15.967149 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:15.967124 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:16.037053 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.037027 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:16.037186 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:16.037121 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:16.037425 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.037407 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:16.037538 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:16.037519 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:16.037596 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.037562 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:16.037645 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:16.037625 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:16.228170 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.228081 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:16.237277 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.237247 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-l2ddq" event={"ID":"bbd7a723-acf0-4c7c-95a5-2bd54d65460d","Type":"ContainerStarted","Data":"40257ad2c9095e40ff3ba640621f3efb2d4584e9a9a5e92a291f4d0cd41c0613"} Oct 30 17:56:16.240379 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.240349 2574 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="fadf7477261abef9ff48a41d3c6a663a6462bf43769a00c7fa227e28c5a8e72c" exitCode=0 Oct 30 17:56:16.240517 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.240435 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerDied","Data":"fadf7477261abef9ff48a41d3c6a663a6462bf43769a00c7fa227e28c5a8e72c"} Oct 30 17:56:16.242561 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.242529 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tg6rw" event={"ID":"f4303592-c23b-49bc-a242-dfefafe43d0a","Type":"ContainerStarted","Data":"00ec75a4adc74db8cba47043c33971603e4ce76a93d0b51cc6ce0ee137aabbad"} Oct 30 17:56:16.245054 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.245030 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" event={"ID":"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4","Type":"ContainerStarted","Data":"296638e1d5002c7fe6af4f4fde0d30200989dafa7158729e31e31e53cc1257d0"} Oct 30 17:56:16.277857 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.277810 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-l2ddq" podStartSLOduration=3.551931774 podStartE2EDuration="11.27779539s" totalImagesPullingTime="7.725863616s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:16.277602257 +0000 UTC m=+10.723941657" watchObservedRunningTime="2026-10-30 17:56:16.27779539 +0000 UTC m=+10.724134796" Oct 30 17:56:16.373349 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.373294 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tg6rw" podStartSLOduration=3.947693128 podStartE2EDuration="11.373275748s" totalImagesPullingTime="7.42558262s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:16.37316026 +0000 UTC m=+10.819499652" watchObservedRunningTime="2026-10-30 17:56:16.373275748 +0000 UTC m=+10.819615137" Oct 30 17:56:16.608568 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.608534 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:16.816611 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:16.816582 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:17.249624 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.249595 2574 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="d741d06c0b470b90a019f20ae0cf6e249cfbc3bd76a1cb3fef2c72eb87a98ca7" exitCode=0 Oct 30 17:56:17.250156 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.249917 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerDied","Data":"d741d06c0b470b90a019f20ae0cf6e249cfbc3bd76a1cb3fef2c72eb87a98ca7"} Oct 30 17:56:17.252778 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.252737 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" event={"ID":"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4","Type":"ContainerStarted","Data":"57bd3869bb6ba356fb854020cfdcb1c854ece1c4f65dbe81dadb3f93562322c8"} Oct 30 17:56:17.255454 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.255427 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-7llnl" event={"ID":"5b91437e-1659-46b1-bc95-687ddd06c392","Type":"ContainerStarted","Data":"b84460d70e6820d9de359ae7e2b06857d55cb6133d2836daa2959b8046e44ac2"} Oct 30 17:56:17.341416 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.341367 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-7llnl" podStartSLOduration=4.808612673 podStartE2EDuration="12.341350669s" totalImagesPullingTime="7.532737996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:17.340721668 +0000 UTC m=+11.787061058" watchObservedRunningTime="2026-10-30 17:56:17.341350669 +0000 UTC m=+11.787690061" Oct 30 17:56:17.423767 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.423735 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:17.673034 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.672988 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:17.971159 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:17.971077 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:18.034128 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.034104 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:18.034289 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.034155 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:18.034289 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.034160 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:18.034289 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:18.034245 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:18.034440 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:18.034337 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:18.034501 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:18.034430 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:18.259043 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.258966 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jhj27" event={"ID":"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790","Type":"ContainerStarted","Data":"ebf7511550d0c8445e31c4929a4a7f7dc3e30770fc29c86b41630474ba841777"} Oct 30 17:56:18.262030 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.261934 2574 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="92545f620a73dd66ad7a2fd9077f9cb513eedb172efb5e31a8f4407847b8f3b7" exitCode=0 Oct 30 17:56:18.262030 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.262017 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerDied","Data":"92545f620a73dd66ad7a2fd9077f9cb513eedb172efb5e31a8f4407847b8f3b7"} Oct 30 17:56:18.303508 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.303447 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" podStartSLOduration=5.639610831 podStartE2EDuration="13.303430837s" totalImagesPullingTime="7.663820006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:17.385430294 +0000 UTC m=+11.831769683" watchObservedRunningTime="2026-10-30 17:56:18.303430837 +0000 UTC m=+12.749770227" Oct 30 17:56:18.303616 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.303595 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jhj27" podStartSLOduration=3.771945342 podStartE2EDuration="13.303585373s" totalImagesPullingTime="9.531640031s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:18.302865112 +0000 UTC m=+12.749204500" watchObservedRunningTime="2026-10-30 17:56:18.303585373 +0000 UTC m=+12.749924764" Oct 30 17:56:18.361507 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.361419 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:18.558482 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:18.558389 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:19.459987 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:19.459760 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:20.034449 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.034417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:20.034642 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.034417 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:20.034642 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:20.034553 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:20.034642 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.034601 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:20.034797 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:20.034737 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:20.034855 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:20.034816 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:20.268885 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.268809 2574 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="e58660cec6434f04f2c0dbde7cf81ff271bd180aa5097c9f746b37673843b408" exitCode=0 Oct 30 17:56:20.268885 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.268860 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerDied","Data":"e58660cec6434f04f2c0dbde7cf81ff271bd180aa5097c9f746b37673843b408"} Oct 30 17:56:20.678946 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.678911 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:20.887036 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:20.887006 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:21.274371 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:21.274341 2574 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="0df97850265b3d6850dc5a026354060e293c3227db38d7c0a3afb9157bcccb3f" exitCode=0 Oct 30 17:56:21.274534 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:21.274380 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerDied","Data":"0df97850265b3d6850dc5a026354060e293c3227db38d7c0a3afb9157bcccb3f"} Oct 30 17:56:21.923612 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:21.923578 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:22.033926 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.033901 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:22.034072 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.033944 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:22.034072 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:22.033994 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:22.034202 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:22.034103 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:22.034202 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.034138 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:22.034285 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:22.034212 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:22.138230 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.138203 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:22.278591 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.278539 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-cpntx" event={"ID":"64fed540-f4cb-4e19-b3d9-7531e486c4f5","Type":"ContainerStarted","Data":"d2b6705f5274601ab121eb3d1d8d2c064ee4f341b4f3e27b45f00adca5632561"} Oct 30 17:56:22.360434 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.360394 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-cpntx" podStartSLOduration=7.847007626 podStartE2EDuration="17.360384517s" totalImagesPullingTime="9.513376891s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:22.360206123 +0000 UTC m=+16.806545512" watchObservedRunningTime="2026-10-30 17:56:22.360384517 +0000 UTC m=+16.806723906" Oct 30 17:56:22.372197 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.372171 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:22.625735 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.625699 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:22.830738 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:22.830701 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:23.284164 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.284128 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"35ff358ac250dbd1128e770657c75c97384455942521e2fc2b4f3901b4824a73"} Oct 30 17:56:23.284823 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.284211 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"d6489ec31641acb5d6025ff7db583947241d71bdea63a9838b0278720cb28778"} Oct 30 17:56:23.284823 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.284258 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"34d3a2c80ffe37f623b0a71202bc75892e531b478c3c4c6fd91e59327f89aef8"} Oct 30 17:56:23.284823 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.284313 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"1680f2c198776637f44a6294b8426829d9517f100d715994909544caacc4c0f3"} Oct 30 17:56:23.284823 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.284361 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"3a933a1a99c27ebbb5cb33515f6d365083ae45e7ea4e502be3cbac93fb7cfc98"} Oct 30 17:56:23.284823 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.284407 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"6e2fc77db5860a4237ab518288f5a51609b85cf2091e046a405a55614c7a0c15"} Oct 30 17:56:23.447791 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.447765 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:23.653585 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.653548 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:23.740007 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.739982 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:23.740149 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.740044 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:23.740149 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.740120 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:23.740261 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.740171 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:23.740261 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.740179 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:39.740163871 +0000 UTC m=+34.186503289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:23.740261 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.740240 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:56:39.740222385 +0000 UTC m=+34.186561759 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:23.840392 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.840367 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:23.840547 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.840532 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:23.840627 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.840552 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:23.840627 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.840565 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:23.840627 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:23.840619 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:56:39.840602665 +0000 UTC m=+34.286942050 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:23.965549 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:23.965496 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:24.033518 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:24.033493 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:24.033518 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:24.033511 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:24.033656 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:24.033573 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:24.033690 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:24.033667 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:24.033801 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:24.033783 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:24.033886 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:24.033869 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:25.009943 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:25.009907 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:25.075247 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:25.075225 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l2ddq_bbd7a723-acf0-4c7c-95a5-2bd54d65460d/dns-node-resolver/0.log" Oct 30 17:56:25.219675 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:25.219616 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:25.290861 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:25.290838 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"c649a766e78d55ccebed7f054ecf6ae0666a7112fa059bfc13bdf96da278e061"} Oct 30 17:56:25.423111 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:25.423076 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:25.994257 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:25.994230 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:26.034955 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:26.034934 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:26.035371 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:26.035017 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:26.035371 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:26.035097 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:26.035371 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:26.035172 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:26.035371 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:26.035186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:26.035371 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:26.035299 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:26.075141 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:26.075123 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rd5c5_9cc87edf-5705-49bc-abe7-7cf921ed75ab/node-ca/0.log" Oct 30 17:56:26.586261 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:26.586234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:26.802391 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:26.802363 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:27.007160 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.007127 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:27.212052 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.211800 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:27.298094 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.298062 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" event={"ID":"9d111489-3f0f-49bb-9de2-f400960f01d7","Type":"ContainerStarted","Data":"d5c0360f916e1ec607831382fd252839cbcaba14a2adf42bba54430af682a31e"} Oct 30 17:56:27.298349 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.298324 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:27.298506 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.298419 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:27.313500 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.313478 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:27.313801 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.313779 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:27.392737 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.392693 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" podStartSLOduration=7.202726911 podStartE2EDuration="21.39268224s" totalImagesPullingTime="14.189955329s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:27.392208475 +0000 UTC m=+21.838547953" watchObservedRunningTime="2026-10-30 17:56:27.39268224 +0000 UTC m=+21.839021628" Oct 30 17:56:27.419052 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.419022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:27.986547 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:27.986516 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:28.033583 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.033556 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:28.033583 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.033585 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:28.033790 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.033669 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:28.033790 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:28.033688 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:28.033790 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:28.033735 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:28.033928 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:28.033810 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:28.211178 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.211150 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:28.300367 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.300317 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:28.598169 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.598144 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:28.801594 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:28.801565 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:29.006562 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.006490 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:29.200275 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.200246 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:29.302482 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.302421 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:29.407504 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.407477 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:29.603286 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.603256 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:29.804468 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.804437 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:29.999825 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:29.999751 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:30.034474 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:30.034437 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:30.034617 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:30.034435 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:30.034617 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:30.034546 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:30.034617 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:30.034569 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:30.034778 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:30.034646 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:30.034778 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:30.034689 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:30.587066 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:30.587040 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:30.790544 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:30.790511 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:31.181453 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:31.181420 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:31.389521 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:31.389490 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:31.595105 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:31.595060 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:31.986053 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:31.985986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.033857 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.033834 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:32.033857 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.033849 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:32.034017 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:32.033940 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:32.034017 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:32.033992 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:32.034017 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.034011 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:32.034126 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:32.034080 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:32.369826 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.369807 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:32.369948 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.369901 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:32.370287 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.370273 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:56:32.582389 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.582367 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.787661 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.787589 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:32.981761 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:32.981729 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:33.152475 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:33.152432 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:56:33.152666 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:33.152651 2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:56:33.161981 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:33.161949 2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" podUID="9d111489-3f0f-49bb-9de2-f400960f01d7" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:56:33.169921 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:33.169895 2574 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" podUID="9d111489-3f0f-49bb-9de2-f400960f01d7" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 17:56:33.394498 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:33.394473 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:33.593368 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:33.593339 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:34.034191 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.034132 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:34.034538 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.034135 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:34.034538 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:34.034230 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:34.034538 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.034142 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:34.034538 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:34.034302 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:34.034765 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:34.034366 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:34.190988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.190959 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:34.389424 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.389394 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:34.766805 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.766734 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:34.964471 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:34.964430 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:35.157826 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:35.157799 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:35.590817 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:35.590789 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:35.794058 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:35.794032 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:35.995236 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:35.995155 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:36.034426 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:36.034397 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:36.034570 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:36.034544 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:36.034570 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:36.034555 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:36.034682 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:36.034640 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:36.034734 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:36.034695 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:36.034797 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:36.034771 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:36.229955 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:36.229922 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:38.034217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:38.034186 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:38.034217 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:38.034212 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:38.034759 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:38.034267 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:38.034759 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:38.034315 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:38.034759 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:38.034413 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:38.034759 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:38.034541 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:38.864930 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:38.864897 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:39.067643 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:39.067612 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:39.318448 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:39.318419 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:39.533428 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:39.533382 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:39.752657 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:39.752587 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:39.752657 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:39.752642 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:39.752846 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.752733 2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:39.752846 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.752776 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret podName:788e0688-d9fb-48af-993d-c3bc6a0bc4c5 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:11.752764388 +0000 UTC m=+66.199103755 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret") pod "global-pull-secret-syncer-gqtbs" (UID: "788e0688-d9fb-48af-993d-c3bc6a0bc4c5") : object "kube-system"/"original-pull-secret" not registered Oct 30 17:56:39.752846 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.752733 2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:39.752846 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.752837 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs podName:3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b nodeName:}" failed. No retries permitted until 2026-10-30 17:57:11.752818179 +0000 UTC m=+66.199157559 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs") pod "network-metrics-daemon-c8fkb" (UID: "3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 17:56:39.853369 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:39.853344 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:39.853489 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.853451 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 17:56:39.853489 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.853483 2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 17:56:39.853583 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.853493 2574 projected.go:196] Error preparing data for projected volume kube-api-access-gp5mb for pod openshift-network-diagnostics/network-check-target-pjfgc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:39.853583 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:39.853543 2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb podName:a6990713-1f8a-4ecc-9279-9e28d2c0cba4 nodeName:}" failed. No retries permitted until 2026-10-30 17:57:11.853529467 +0000 UTC m=+66.299868846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gp5mb" (UniqueName: "kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb") pod "network-check-target-pjfgc" (UID: "a6990713-1f8a-4ecc-9279-9e28d2c0cba4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 17:56:40.034357 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:40.034286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:40.034357 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:40.034305 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:40.034533 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:40.034286 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:40.034533 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:40.034399 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:40.034533 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:40.034446 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:40.034639 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:40.034552 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:42.034416 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:42.034381 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:42.034824 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:42.034445 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:42.034824 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:42.034446 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:42.034824 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:42.034564 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:42.034824 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:42.034613 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:42.034824 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:42.034672 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:44.033678 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:44.033647 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:44.033678 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:44.033675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:44.034113 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:44.033650 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:44.034113 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:44.033765 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:44.034113 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:44.033827 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:44.034113 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:44.033879 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:45.567664 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:45.567624 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:45.775277 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:45.775234 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:46.034983 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:46.034910 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:46.035135 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:46.035003 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:46.035135 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:46.035070 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:46.035257 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:46.035149 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:46.035257 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:46.035195 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:46.035335 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:46.035266 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:46.674836 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:46.674801 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:46.965093 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:46.965022 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:47.179204 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.179173 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:47.537579 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.537543 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:47.974044 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.973815 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqtbs"] Oct 30 17:56:47.974334 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.974124 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjfgc"] Oct 30 17:56:47.974334 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.974208 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:47.974334 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:47.974307 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:47.974568 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.974551 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:47.974655 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:47.974639 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:47.975875 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.975856 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c8fkb"] Oct 30 17:56:47.975966 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:47.975953 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:47.976068 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:47.976052 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:48.056522 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:48.056494 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:48.255995 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:48.255934 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:48.465517 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:48.465488 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:48.686394 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:48.686350 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:49.458575 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:49.458495 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:49.667082 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:49.667057 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:49.877016 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:49.876991 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:50.033937 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.033909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:50.033937 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.033933 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:50.034151 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.033912 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:50.034151 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:50.034005 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:50.034151 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:50.034094 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:50.034246 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:50.034168 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:50.106376 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.106344 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:50.407016 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.406986 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:50.614829 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.614793 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:50.810035 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:50.809962 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:51.016576 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:51.016546 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:51.182698 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:51.182676 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l2ddq_bbd7a723-acf0-4c7c-95a5-2bd54d65460d/dns-node-resolver/0.log" Oct 30 17:56:51.621406 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:51.621372 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:51.827745 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:51.827707 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:52.034355 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.034333 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:52.034355 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.034347 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:52.034556 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:52.034440 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqtbs" podUID="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" Oct 30 17:56:52.035880 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:52.034841 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-c8fkb" podUID="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" Oct 30 17:56:52.035880 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.034879 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:52.035880 ip-10-0-136-241 kubenswrapper[2574]: E1030 17:56:52.034996 2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-pjfgc" podUID="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" Oct 30 17:56:52.333544 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.333517 2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeReady" Oct 30 17:56:52.333668 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.333638 2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 17:56:52.382067 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.382044 2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rd5c5_9cc87edf-5705-49bc-abe7-7cf921ed75ab/node-ca/0.log" Oct 30 17:56:52.397194 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.397164 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vm92t"] Oct 30 17:56:52.398926 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.398909 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.399973 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.399955 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-56qxs"] Oct 30 17:56:52.401690 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.401675 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.419821 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.419805 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:56:52.420085 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.420070 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:56:52.420334 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.420322 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:56:52.423138 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.423114 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:56:52.423233 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.423139 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7j6rv\"" Oct 30 17:56:52.423304 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.423270 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:56:52.423363 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.423328 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:56:52.424787 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.424095 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-z5c26"] Oct 30 17:56:52.425713 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.425696 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.433364 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.433347 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vm92t"] Oct 30 17:56:52.433896 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.433881 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56qxs"] Oct 30 17:56:52.453003 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.452983 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5c26"] Oct 30 17:56:52.497706 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.497687 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-fb8fd8fd8-5ks86"] Oct 30 17:56:52.499326 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.499313 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.501830 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.501810 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:52.532521 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.532493 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:56:52.532729 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.532712 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2dl6z\"" Oct 30 17:56:52.532833 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.532821 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dqs6z\"" Oct 30 17:56:52.532874 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.532831 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:56:52.532966 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.532955 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:56:52.533101 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533082 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:56:52.533101 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533088 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:56:52.533242 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533213 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:56:52.533363 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533352 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:56:52.533501 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533489 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:56:52.533542 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533489 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksmnn\"" Oct 30 17:56:52.533630 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533612 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:56:52.533663 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.533648 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:56:52.547844 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.547827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/699b6cb3-446f-440e-8d68-66c4e09ee322-metrics-tls\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.547892 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.547851 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q4kz\" (UniqueName: \"kubernetes.io/projected/bc0cafd7-e383-4391-8d85-073289c3f442-kube-api-access-7q4kz\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.547892 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.547873 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/699b6cb3-446f-440e-8d68-66c4e09ee322-config-volume\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.547892 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.547888 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699b6cb3-446f-440e-8d68-66c4e09ee322-tmp-dir\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.548036 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548018 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rqz\" (UniqueName: \"kubernetes.io/projected/699b6cb3-446f-440e-8d68-66c4e09ee322-kube-api-access-v8rqz\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.548089 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0cafd7-e383-4391-8d85-073289c3f442-cert\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.548089 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548084 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.548156 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548101 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8mhq\" (UniqueName: \"kubernetes.io/projected/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-api-access-m8mhq\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.548156 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548124 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.548226 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548200 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-data-volume\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.548292 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.548274 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-crio-socket\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.555385 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.555361 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb8fd8fd8-5ks86"] Oct 30 17:56:52.649078 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649027 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.649078 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649059 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-crio-socket\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649100 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649169 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/699b6cb3-446f-440e-8d68-66c4e09ee322-metrics-tls\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649218 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4kz\" (UniqueName: \"kubernetes.io/projected/bc0cafd7-e383-4391-8d85-073289c3f442-kube-api-access-7q4kz\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649243 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649245 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-crio-socket\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649313 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649368 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/699b6cb3-446f-440e-8d68-66c4e09ee322-config-volume\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649403 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699b6cb3-446f-440e-8d68-66c4e09ee322-tmp-dir\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649428 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.649452 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649453 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rqz\" (UniqueName: \"kubernetes.io/projected/699b6cb3-446f-440e-8d68-66c4e09ee322-kube-api-access-v8rqz\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649492 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0cafd7-e383-4391-8d85-073289c3f442-cert\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649522 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649635 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mhq\" (UniqueName: \"kubernetes.io/projected/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-api-access-m8mhq\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649710 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649770 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-data-volume\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649776 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699b6cb3-446f-440e-8d68-66c4e09ee322-tmp-dir\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.649977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.649797 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.650261 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.650043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/699b6cb3-446f-440e-8d68-66c4e09ee322-config-volume\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.650261 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.650138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-data-volume\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.650368 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.650348 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.653168 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.653138 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/699b6cb3-446f-440e-8d68-66c4e09ee322-metrics-tls\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.653168 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.653151 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0cafd7-e383-4391-8d85-073289c3f442-cert\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.653168 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.653146 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.673271 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.673246 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rqz\" (UniqueName: \"kubernetes.io/projected/699b6cb3-446f-440e-8d68-66c4e09ee322-kube-api-access-v8rqz\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.673271 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.673267 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mhq\" (UniqueName: \"kubernetes.io/projected/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-api-access-m8mhq\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.673385 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.673364 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4kz\" (UniqueName: \"kubernetes.io/projected/bc0cafd7-e383-4391-8d85-073289c3f442-kube-api-access-7q4kz\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.710298 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.710267 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:56:52.717701 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.717685 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:56:52.724633 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.724605 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:52.737291 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.737268 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:52.751057 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751036 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.751150 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751094 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.751150 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751113 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.751150 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751129 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.751150 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751144 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.751274 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751176 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.751274 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751191 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.752000 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.751950 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.752175 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.752151 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.752239 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.752193 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.752239 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.752199 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.759061 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.756860 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.759061 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.758924 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.779272 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.779240 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:52.808568 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:52.808245 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:56:53.025007 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.024949 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vm92t"] Oct 30 17:56:53.028803 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:53.028776 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc0cafd7_e383_4391_8d85_073289c3f442.slice/crio-994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19 WatchSource:0}: Error finding container 994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19: Status 404 returned error can't find the container with id 994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19 Oct 30 17:56:53.037474 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.037360 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-56qxs"] Oct 30 17:56:53.041598 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:53.041566 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85b2e7b1_1141_4cb4_a0fa_5376cf8014d8.slice/crio-598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3 WatchSource:0}: Error finding container 598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3: Status 404 returned error can't find the container with id 598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3 Oct 30 17:56:53.059700 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.059678 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-z5c26"] Oct 30 17:56:53.062782 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:53.062759 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699b6cb3_446f_440e_8d68_66c4e09ee322.slice/crio-75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc WatchSource:0}: Error finding container 75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc: Status 404 returned error can't find the container with id 75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc Oct 30 17:56:53.108348 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.108328 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-fb8fd8fd8-5ks86"] Oct 30 17:56:53.111471 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:53.111433 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4652dca2_9b46_43e6_87d4_70c094ecf09a.slice/crio-37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864 WatchSource:0}: Error finding container 37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864: Status 404 returned error can't find the container with id 37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864 Oct 30 17:56:53.112583 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.112557 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:53.348198 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.348171 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vm92t" event={"ID":"bc0cafd7-e383-4391-8d85-073289c3f442","Type":"ContainerStarted","Data":"994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19"} Oct 30 17:56:53.349200 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.349179 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb8fd8fd8-5ks86" event={"ID":"4652dca2-9b46-43e6-87d4-70c094ecf09a","Type":"ContainerStarted","Data":"37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864"} Oct 30 17:56:53.350255 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.350236 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5c26" event={"ID":"699b6cb3-446f-440e-8d68-66c4e09ee322","Type":"ContainerStarted","Data":"75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc"} Oct 30 17:56:53.351613 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.351594 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56qxs" event={"ID":"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8","Type":"ContainerStarted","Data":"fa08f72ecd3cdc2a461217803af69544fb4cc0a00fafe675bbac72cc67ab2640"} Oct 30 17:56:53.351684 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.351633 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56qxs" event={"ID":"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8","Type":"ContainerStarted","Data":"598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3"} Oct 30 17:56:53.770674 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.770596 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:53.989722 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:53.989622 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:54.034567 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.034534 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:56:54.034783 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.034764 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:56:54.035441 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.034475 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:56:54.050526 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.050502 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:56:54.050635 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.050564 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:56:54.063736 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.063544 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzlmk\"" Oct 30 17:56:54.063736 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.063634 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dxtgs\"" Oct 30 17:56:54.063885 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.063790 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:56:54.063885 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.063858 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:56:54.063961 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.063905 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkbt2\"" Oct 30 17:56:54.358738 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.358702 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56qxs" event={"ID":"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8","Type":"ContainerStarted","Data":"3c98ed63f04659b31fe607b993fe6c928396fac21098bfa0e0e49a6a5363279f"} Oct 30 17:56:54.895296 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:54.895264 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:55.081539 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.081505 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq"] Oct 30 17:56:55.083902 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.083877 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.102128 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.102106 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:56:55.114215 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.114192 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dhnvk\"" Oct 30 17:56:55.114313 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.114196 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 17:56:55.114313 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.114238 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:56:55.114450 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.114336 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:56:55.114450 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.114402 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:56:55.115356 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.115292 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq"] Oct 30 17:56:55.121185 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.121050 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:55.132710 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.132690 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv"] Oct 30 17:56:55.144792 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.143738 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.150420 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.150368 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4fgr8"] Oct 30 17:56:55.152487 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.152450 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.161712 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.161693 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv"] Oct 30 17:56:55.172011 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.171733 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.172011 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.171788 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.172011 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.171842 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185b41a3-2575-4865-ba56-77b1c1b40989-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.172011 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.171892 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwnl8\" (UniqueName: \"kubernetes.io/projected/185b41a3-2575-4865-ba56-77b1c1b40989-kube-api-access-qwnl8\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.195751 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.195701 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:56:55.195882 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.195811 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 17:56:55.201691 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.201667 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:56:55.201842 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.201826 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:56:55.202047 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.202023 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:56:55.202130 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.202115 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cclgx\"" Oct 30 17:56:55.202209 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.202195 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 17:56:55.202300 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.202286 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c2g6c\"" Oct 30 17:56:55.272854 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.272823 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-tls\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.272950 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.272870 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.272950 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.272925 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnl8\" (UniqueName: \"kubernetes.io/projected/185b41a3-2575-4865-ba56-77b1c1b40989-kube-api-access-qwnl8\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.273060 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.272956 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb1042dd-c449-4c9e-9a56-899e68938400-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.273060 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273013 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.273060 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273044 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-textfile\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273202 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273072 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-wtmp\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273202 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273098 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273202 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273125 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp6xh\" (UniqueName: \"kubernetes.io/projected/eb1042dd-c449-4c9e-9a56-899e68938400-kube-api-access-vp6xh\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.273202 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273148 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-root\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273212 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9szl\" (UniqueName: \"kubernetes.io/projected/92f93306-6430-45d2-887a-f056426c57ed-kube-api-access-g9szl\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273240 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-sys\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273259 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-metrics-client-ca\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273275 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273295 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273315 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273335 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273355 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.273383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273374 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185b41a3-2575-4865-ba56-77b1c1b40989-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.273957 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.273938 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185b41a3-2575-4865-ba56-77b1c1b40989-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.276867 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.276848 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.285139 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.285110 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.303329 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.303298 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnl8\" (UniqueName: \"kubernetes.io/projected/185b41a3-2575-4865-ba56-77b1c1b40989-kube-api-access-qwnl8\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.322361 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.322280 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373693 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373755 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-tls\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373783 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373812 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb1042dd-c449-4c9e-9a56-899e68938400-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373858 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-textfile\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373884 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-wtmp\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373908 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373933 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6xh\" (UniqueName: \"kubernetes.io/projected/eb1042dd-c449-4c9e-9a56-899e68938400-kube-api-access-vp6xh\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.373961 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-root\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374025 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9szl\" (UniqueName: \"kubernetes.io/projected/92f93306-6430-45d2-887a-f056426c57ed-kube-api-access-g9szl\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374054 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-sys\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374076 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-metrics-client-ca\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374104 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374135 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374160 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.379717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.374782 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.375409 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.376410 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-sys\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.377039 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.377680 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-textfile\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.377869 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb1042dd-c449-4c9e-9a56-899e68938400-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.378346 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-wtmp\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.378597 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-root\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.380604 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.378719 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-metrics-client-ca\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.391092 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.388040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.394971 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.392829 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-tls\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.396190 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.396164 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.397037 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.397007 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.400246 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.400223 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:56:55.401831 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.401778 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6xh\" (UniqueName: \"kubernetes.io/projected/eb1042dd-c449-4c9e-9a56-899e68938400-kube-api-access-vp6xh\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.404088 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.404068 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9szl\" (UniqueName: \"kubernetes.io/projected/92f93306-6430-45d2-887a-f056426c57ed-kube-api-access-g9szl\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.471927 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.471517 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:56:55.478749 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.478723 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:56:55.523138 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.523106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:55.721564 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.721488 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq"] Oct 30 17:56:55.725272 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:55.725238 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod185b41a3_2575_4865_ba56_77b1c1b40989.slice/crio-cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4 WatchSource:0}: Error finding container cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4: Status 404 returned error can't find the container with id cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4 Oct 30 17:56:55.731909 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.731781 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:55.812523 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:55.812422 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv"] Oct 30 17:56:56.369249 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.369130 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" event={"ID":"eb1042dd-c449-4c9e-9a56-899e68938400","Type":"ContainerStarted","Data":"731a4a90ddf12d02def07224fa394c694121fc7436bfbadaa2ef9ef5e3312cc5"} Oct 30 17:56:56.376762 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.376714 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:56.378943 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.378123 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5c26" event={"ID":"699b6cb3-446f-440e-8d68-66c4e09ee322","Type":"ContainerStarted","Data":"0070d97ec4312fc20c1e6ffc48627c9171a0eea5fdf46f17247ff4b2d235d6ad"} Oct 30 17:56:56.378943 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.378182 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-z5c26" event={"ID":"699b6cb3-446f-440e-8d68-66c4e09ee322","Type":"ContainerStarted","Data":"a1be596d04bb4d2c17f0f55ed6577b01ca0ef6b861363b11968d9e150d5f3d19"} Oct 30 17:56:56.378943 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.378833 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-z5c26" Oct 30 17:56:56.382897 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.382451 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" event={"ID":"185b41a3-2575-4865-ba56-77b1c1b40989","Type":"ContainerStarted","Data":"ac886f2ac4ddb6db33d1d4e54ffde493f918362e4d5f132740cfd5bffc1681f8"} Oct 30 17:56:56.382897 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.382517 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" event={"ID":"185b41a3-2575-4865-ba56-77b1c1b40989","Type":"ContainerStarted","Data":"d4374b4ac29958200db406197dc85d208741f12c28b768cbeeaa5d0e856bb9c6"} Oct 30 17:56:56.382897 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.382546 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" event={"ID":"185b41a3-2575-4865-ba56-77b1c1b40989","Type":"ContainerStarted","Data":"cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4"} Oct 30 17:56:56.387616 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.387382 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-56qxs" event={"ID":"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8","Type":"ContainerStarted","Data":"c6c72179035e1c07b3d87fb05d5ab452a57ebda2cdf20c8141bdcbb29d8b53f4"} Oct 30 17:56:56.389391 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.389327 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fgr8" event={"ID":"92f93306-6430-45d2-887a-f056426c57ed","Type":"ContainerStarted","Data":"e7a9612b393a7e97752ef9e32e95c167166ead35445eb6f1636e84ec8987a386"} Oct 30 17:56:56.541602 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.541120 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-56qxs" podStartSLOduration=2.22845208 podStartE2EDuration="4.541104743s" totalImagesPullingTime="2.312652663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:56.540042602 +0000 UTC m=+50.986381992" watchObservedRunningTime="2026-10-30 17:56:56.541104743 +0000 UTC m=+50.987444131" Oct 30 17:56:56.586678 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.586626 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-z5c26" podStartSLOduration=2.397913765 podStartE2EDuration="4.586607059s" totalImagesPullingTime="2.188693294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:56.585969633 +0000 UTC m=+51.032309022" watchObservedRunningTime="2026-10-30 17:56:56.586607059 +0000 UTC m=+51.032946449" Oct 30 17:56:56.601512 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.601482 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:56.641726 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.641698 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:56:56.651078 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.648567 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.682740 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.682946 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.683033 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.683136 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.683200 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.683289 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:56:56.683559 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.683402 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:56:56.684554 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.684530 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:56:56.684698 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.684602 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t7pkc\"" Oct 30 17:56:56.684698 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.684639 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:56:56.686758 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.686713 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:56:56.785977 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.785937 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786081 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.785990 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786081 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786022 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786081 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786076 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786189 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786189 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786127 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786189 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786155 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786279 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786186 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786279 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786234 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786279 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786258 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786370 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786294 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786370 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786322 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.786370 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.786348 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.837694 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.837627 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887089 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887133 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887165 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887195 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887224 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887316 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887342 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887389 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887415 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887438 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887478 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.887511 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.888400 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.889524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.889065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.890345 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.890083 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.902104 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.902055 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.902775 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.902737 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.902889 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.902781 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.904414 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.904311 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.904414 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.904346 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.904414 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.904370 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.905084 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.904710 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.905084 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.905042 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.905539 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.905454 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.918524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.918480 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:56.973381 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:56.973346 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:56:57.130740 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.130694 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:57.311715 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.311690 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:56:57.315573 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:56:57.315548 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa838616_f66c_424f_bbd8_eab8cc77ea9c.slice/crio-2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5 WatchSource:0}: Error finding container 2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5: Status 404 returned error can't find the container with id 2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5 Oct 30 17:56:57.339482 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.339170 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:57.394566 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.394102 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vm92t" event={"ID":"bc0cafd7-e383-4391-8d85-073289c3f442","Type":"ContainerStarted","Data":"785cbacc410f45de2938b2740c5e1c0fc84aad0447b2015beada7d00539a64e7"} Oct 30 17:56:57.395527 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.395508 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-67bc75cc55-5scfq"] Oct 30 17:56:57.398006 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.397975 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" event={"ID":"eb1042dd-c449-4c9e-9a56-899e68938400","Type":"ContainerStarted","Data":"43ca767ea9997121327f487928b136797db9add27ac59910dcdc353dfcba3cf9"} Oct 30 17:56:57.398103 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.398027 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb8fd8fd8-5ks86" event={"ID":"4652dca2-9b46-43e6-87d4-70c094ecf09a","Type":"ContainerStarted","Data":"f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e"} Oct 30 17:56:57.398103 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.398065 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.400454 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.400431 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" event={"ID":"185b41a3-2575-4865-ba56-77b1c1b40989","Type":"ContainerStarted","Data":"d9c0726ab48fecd3bb95c0a68a45d20d560e77d6a6e1aac8e7bfd23973327eb3"} Oct 30 17:56:57.401946 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.401924 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5"} Oct 30 17:56:57.403692 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.403667 2574 generic.go:356] "Generic (PLEG): container finished" podID="92f93306-6430-45d2-887a-f056426c57ed" containerID="dfc9ae5a990c8217fcf69369b3115e54d6029bb134d016afe4f5ec4d69236026" exitCode=0 Oct 30 17:56:57.403782 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.403726 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fgr8" event={"ID":"92f93306-6430-45d2-887a-f056426c57ed","Type":"ContainerDied","Data":"dfc9ae5a990c8217fcf69369b3115e54d6029bb134d016afe4f5ec4d69236026"} Oct 30 17:56:57.413432 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.413412 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 17:56:57.413622 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.413417 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 17:56:57.413702 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.413624 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 17:56:57.415618 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.415568 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 17:56:57.416438 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.416414 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5zvv9\"" Oct 30 17:56:57.427179 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.427157 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67bc75cc55-5scfq"] Oct 30 17:56:57.437242 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.437169 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vm92t" podStartSLOduration=1.8872617109999998 podStartE2EDuration="5.437153096s" totalImagesPullingTime="3.549891385s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:57.436503464 +0000 UTC m=+51.882842854" watchObservedRunningTime="2026-10-30 17:56:57.437153096 +0000 UTC m=+51.883492487" Oct 30 17:56:57.491976 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.491686 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.491976 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.491919 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.491976 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.491948 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-grpc-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.492135 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.492010 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.492135 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.492103 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.492196 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.492146 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d35b6637-df31-4561-9b9e-ebf305bd997f-tmp\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.493846 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.493827 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d35b6637-df31-4561-9b9e-ebf305bd997f-metrics-client-ca\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.494071 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.494047 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.494127 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.494110 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfv4g\" (UniqueName: \"kubernetes.io/projected/d35b6637-df31-4561-9b9e-ebf305bd997f-kube-api-access-wfv4g\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.513661 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.513641 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 17:56:57.513802 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.513718 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fc3pb24n385dj\"" Oct 30 17:56:57.582921 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.582854 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-fb8fd8fd8-5ks86" podStartSLOduration=1.6618868610000002 podStartE2EDuration="5.582836173s" totalImagesPullingTime="3.920949312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:57.582812285 +0000 UTC m=+52.029151675" watchObservedRunningTime="2026-10-30 17:56:57.582836173 +0000 UTC m=+52.029175564" Oct 30 17:56:57.595097 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595058 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595200 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595106 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-grpc-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595200 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595140 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595200 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595188 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595336 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595230 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d35b6637-df31-4561-9b9e-ebf305bd997f-tmp\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595336 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595261 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d35b6637-df31-4561-9b9e-ebf305bd997f-metrics-client-ca\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595336 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595297 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595336 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595321 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfv4g\" (UniqueName: \"kubernetes.io/projected/d35b6637-df31-4561-9b9e-ebf305bd997f-kube-api-access-wfv4g\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.595522 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.595375 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.596524 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.596497 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d35b6637-df31-4561-9b9e-ebf305bd997f-tmp\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.596648 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.596528 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d35b6637-df31-4561-9b9e-ebf305bd997f-metrics-client-ca\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.598062 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.598040 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.598285 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.598260 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.598348 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.598302 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.599184 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.599152 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.599309 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.599295 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-grpc-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.599384 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.599368 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.619791 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.619771 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfv4g\" (UniqueName: \"kubernetes.io/projected/d35b6637-df31-4561-9b9e-ebf305bd997f-kube-api-access-wfv4g\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.626021 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.625984 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" podStartSLOduration=1.448725374 podStartE2EDuration="2.625973858s" totalImagesPullingTime="1.177248484s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:57.625897737 +0000 UTC m=+52.072237127" watchObservedRunningTime="2026-10-30 17:56:57.625973858 +0000 UTC m=+52.072313246" Oct 30 17:56:57.718621 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.718590 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:56:57.776658 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.776616 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:57.986979 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:57.986902 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:58.030488 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.030444 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-67bc75cc55-5scfq"] Oct 30 17:56:58.189335 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.189281 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:58.409135 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.409103 2574 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" exitCode=0 Oct 30 17:56:58.409505 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.409186 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} Oct 30 17:56:58.411413 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.411394 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fgr8" event={"ID":"92f93306-6430-45d2-887a-f056426c57ed","Type":"ContainerStarted","Data":"07d8ab789bb39a5320ff403ab01233386926bb102cb765309025c9823c83351a"} Oct 30 17:56:58.411535 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.411438 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4fgr8" event={"ID":"92f93306-6430-45d2-887a-f056426c57ed","Type":"ContainerStarted","Data":"e713539d7b4ab5e7ba2290f67facd17503af9eae94f01ba3bcb4200ec76fa5c9"} Oct 30 17:56:58.412575 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.412545 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"89a4d82ffb50b918a668728c562d5ccc314756d8025be54a609c96c44f548f3c"} Oct 30 17:56:58.414532 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.414509 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" event={"ID":"eb1042dd-c449-4c9e-9a56-899e68938400","Type":"ContainerStarted","Data":"9f9d29c844bf096ddcc83d93f73c08e970bf621dbc4d3f19698785e07ccff955"} Oct 30 17:56:58.414624 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.414560 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" event={"ID":"eb1042dd-c449-4c9e-9a56-899e68938400","Type":"ContainerStarted","Data":"1ce2b988eb74401c8d48270e65dc3e60caa1ada9962d02eef4dd317c3b251143"} Oct 30 17:56:58.486787 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.486754 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:58.519383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.519297 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" podStartSLOduration=2.056604502 podStartE2EDuration="3.519279244s" totalImagesPullingTime="1.462674742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:58.516054168 +0000 UTC m=+52.962393553" watchObservedRunningTime="2026-10-30 17:56:58.519279244 +0000 UTC m=+52.965618635" Oct 30 17:56:58.561145 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.561102 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4fgr8" podStartSLOduration=2.432610425 podStartE2EDuration="3.561089064s" totalImagesPullingTime="1.042570214s" totalInitContainerRuntime="85.908425ms" isStatefulPod=true podCreationTimestamp="2026-10-30 17:56:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:56:58.559928672 +0000 UTC m=+53.006268061" watchObservedRunningTime="2026-10-30 17:56:58.561089064 +0000 UTC m=+53.007428454" Oct 30 17:56:58.707834 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.707798 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:58.908337 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:58.908303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:59.619847 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.619813 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf"] Oct 30 17:56:59.622370 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.622348 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.637810 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.637787 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 17:56:59.652704 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.650330 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d6gl7\"" Oct 30 17:56:59.652704 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.650410 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:56:59.652704 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.650449 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 17:56:59.652704 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.652452 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf"] Oct 30 17:56:59.652704 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.652599 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fbpgnt1o3ru57\"" Oct 30 17:56:59.652704 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.652664 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 17:56:59.717742 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717706 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.717877 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717750 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cd87de0b-e92c-4740-ab77-b5793a227d40-audit-log\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.717877 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717795 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-client-certs\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.717877 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717836 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5s8\" (UniqueName: \"kubernetes.io/projected/cd87de0b-e92c-4740-ab77-b5793a227d40-kube-api-access-4r5s8\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.717877 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717862 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-tls\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.718071 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717899 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-client-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.718071 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.717959 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-metrics-server-audit-profiles\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819215 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819262 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cd87de0b-e92c-4740-ab77-b5793a227d40-audit-log\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819299 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-client-certs\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819333 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5s8\" (UniqueName: \"kubernetes.io/projected/cd87de0b-e92c-4740-ab77-b5793a227d40-kube-api-access-4r5s8\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819352 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-tls\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819377 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-client-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819429 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-metrics-server-audit-profiles\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.819838 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cd87de0b-e92c-4740-ab77-b5793a227d40-audit-log\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.820095 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.820486 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.820417 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-metrics-server-audit-profiles\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.822508 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.822482 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-client-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.822636 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.822610 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-tls\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.823114 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.823090 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-client-certs\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.844158 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.844132 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5s8\" (UniqueName: \"kubernetes.io/projected/cd87de0b-e92c-4740-ab77-b5793a227d40-kube-api-access-4r5s8\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.847703 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.847676 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:56:59.934244 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.934168 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:56:59.997387 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:56:59.997359 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p"] Oct 30 17:57:00.000096 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.000075 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.016994 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.016949 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 17:57:00.017311 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.017295 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9zbdg\"" Oct 30 17:57:00.019551 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.019486 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p"] Oct 30 17:57:00.049288 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.049261 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:00.129728 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.129708 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/447f1a32-218e-4599-9a99-011281767a95-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.129831 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.129762 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtg4z\" (UniqueName: \"kubernetes.io/projected/447f1a32-218e-4599-9a99-011281767a95-kube-api-access-wtg4z\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.230696 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.230675 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtg4z\" (UniqueName: \"kubernetes.io/projected/447f1a32-218e-4599-9a99-011281767a95-kube-api-access-wtg4z\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.230788 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.230757 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/447f1a32-218e-4599-9a99-011281767a95-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.233066 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.233041 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/447f1a32-218e-4599-9a99-011281767a95-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.254518 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.254442 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtg4z\" (UniqueName: \"kubernetes.io/projected/447f1a32-218e-4599-9a99-011281767a95-kube-api-access-wtg4z\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.255404 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.255330 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf"] Oct 30 17:57:00.258828 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:00.258804 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd87de0b_e92c_4740_ab77_b5793a227d40.slice/crio-01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629 WatchSource:0}: Error finding container 01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629: Status 404 returned error can't find the container with id 01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629 Oct 30 17:57:00.259126 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.259106 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:00.312400 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.312380 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:00.424371 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.424340 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} Oct 30 17:57:00.424483 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.424430 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} Oct 30 17:57:00.424523 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.424483 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} Oct 30 17:57:00.424523 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.424508 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} Oct 30 17:57:00.424597 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.424530 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} Oct 30 17:57:00.426325 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.426297 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"92163c308fa8665cbe4d9532e9587f300340bbf7251f1f7c9babe2808cb48680"} Oct 30 17:57:00.426432 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.426355 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"ac5ac0a87d07c63596ed16feaf1edf27c4423fb01053d53b80caef74141bb5fa"} Oct 30 17:57:00.426432 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.426398 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"eccbfc224c36b3e0a39ca3ec9f63706a91b69723cda604cc2b415f7e0089caec"} Oct 30 17:57:00.427386 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.427365 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" event={"ID":"cd87de0b-e92c-4740-ab77-b5793a227d40","Type":"ContainerStarted","Data":"01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629"} Oct 30 17:57:00.479910 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.479841 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:00.610279 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:00.610235 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p"] Oct 30 17:57:00.614094 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:00.614069 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod447f1a32_218e_4599_9a99_011281767a95.slice/crio-a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef WatchSource:0}: Error finding container a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef: Status 404 returned error can't find the container with id a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef Oct 30 17:57:01.438699 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.438663 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"341701da6ac2b805cf10473c3c097b185e25c25dd764c4b1842f2a94b6084365"} Oct 30 17:57:01.439189 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.438732 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"6e96a4ebb9ad45c9be7460d2c40e3aa68e0fa9d43d0ba276d19a8ec1f5ca4c69"} Oct 30 17:57:01.439189 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.438835 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:01.439189 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.438909 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" event={"ID":"d35b6637-df31-4561-9b9e-ebf305bd997f","Type":"ContainerStarted","Data":"50d44c546d24887f46f19e6423b8c65860bede3bd6c6c0e5b89aafdbc392a23d"} Oct 30 17:57:01.440679 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.440649 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" event={"ID":"447f1a32-218e-4599-9a99-011281767a95","Type":"ContainerStarted","Data":"a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef"} Oct 30 17:57:01.444784 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.444758 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerStarted","Data":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} Oct 30 17:57:01.491544 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.491504 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" podStartSLOduration=1.914092259 podStartE2EDuration="4.491487611s" totalImagesPullingTime="2.577395352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:57 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:01.489673241 +0000 UTC m=+55.936012632" watchObservedRunningTime="2026-10-30 17:57:01.491487611 +0000 UTC m=+55.937827002" Oct 30 17:57:01.545749 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.545691 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.291197675 podStartE2EDuration="5.545673089s" totalImagesPullingTime="3.242713476s" totalInitContainerRuntime="11.761938ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:01.544440851 +0000 UTC m=+55.990780241" watchObservedRunningTime="2026-10-30 17:57:01.545673089 +0000 UTC m=+55.992012479" Oct 30 17:57:01.785039 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:01.785006 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:02.216075 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.215998 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:02.450480 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.450420 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" event={"ID":"447f1a32-218e-4599-9a99-011281767a95","Type":"ContainerStarted","Data":"8c0dfa5190081ea5e7405c7d803725f963bc65a31b3a180f67fd8f2e294406b9"} Oct 30 17:57:02.450948 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.450528 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:02.452223 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.452194 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" event={"ID":"cd87de0b-e92c-4740-ab77-b5793a227d40","Type":"ContainerStarted","Data":"4a4a14784e109f619c40d2adad5b1bc09ed6a91549bee79c233496db250575ca"} Oct 30 17:57:02.456140 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.456121 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:02.504736 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.504641 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" podStartSLOduration=2.153510137 podStartE2EDuration="3.504627067s" totalImagesPullingTime="1.35111693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:02.503592337 +0000 UTC m=+56.949931737" watchObservedRunningTime="2026-10-30 17:57:02.504627067 +0000 UTC m=+56.950966453" Oct 30 17:57:02.547947 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.547892 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" podStartSLOduration=2.16228181 podStartE2EDuration="3.547877989s" totalImagesPullingTime="1.385596179s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:02.546723092 +0000 UTC m=+56.993062482" watchObservedRunningTime="2026-10-30 17:57:02.547877989 +0000 UTC m=+56.994217377" Oct 30 17:57:02.809013 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.808937 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:02.809013 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.808997 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:02.813717 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:02.813696 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:03.171882 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.171858 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:03.459720 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.459658 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:03.475665 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.475641 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:03.707700 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.707667 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:03.718607 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.718555 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6ccccdd4d8-bz6n6"] Oct 30 17:57:03.721829 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.721812 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.751546 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.751517 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 17:57:03.752500 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.752483 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ccccdd4d8-bz6n6"] Oct 30 17:57:03.766819 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.766793 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.766898 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.766822 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.766898 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.766869 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.766983 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.766903 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.766983 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.766941 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.767099 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.767086 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.767140 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.767115 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.767140 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.767134 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868172 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868146 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868252 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868184 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868252 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868211 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868355 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868266 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868355 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868292 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868355 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868343 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868515 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868365 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868515 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868406 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.868814 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.868790 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.869048 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.869027 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.869126 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.869065 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.869201 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.869181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.869661 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.869644 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.870785 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.870765 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.870862 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.870816 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:03.892045 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:03.892022 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:04.032810 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:04.032788 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:04.336804 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:04.336772 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6ccccdd4d8-bz6n6"] Oct 30 17:57:04.339635 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:04.339610 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d95db0_3d14_46b9_99d5_902c2f740585.slice/crio-7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e WatchSource:0}: Error finding container 7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e: Status 404 returned error can't find the container with id 7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e Oct 30 17:57:04.460945 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:04.460909 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccccdd4d8-bz6n6" event={"ID":"a2d95db0-3d14-46b9-99d5-902c2f740585","Type":"ContainerStarted","Data":"0f95edba0d5169a6de26d1a5179e324e3a71ec10540e836d9a1055df00358607"} Oct 30 17:57:04.461344 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:04.460964 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccccdd4d8-bz6n6" event={"ID":"a2d95db0-3d14-46b9-99d5-902c2f740585","Type":"ContainerStarted","Data":"7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e"} Oct 30 17:57:04.507054 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:04.507009 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6ccccdd4d8-bz6n6" podStartSLOduration=1.506997869 podStartE2EDuration="1.506997869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:04.506318706 +0000 UTC m=+58.952658095" watchObservedRunningTime="2026-10-30 17:57:04.506997869 +0000 UTC m=+58.953337258" Oct 30 17:57:04.833069 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:04.833031 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:05.042036 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:05.041999 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:05.596771 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:05.596741 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:05.797338 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:05.797303 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:06.485810 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.485777 2574 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ccccdd4d8-bz6n6"] Oct 30 17:57:06.524090 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.524063 2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-f9d75c56b-xtlgz"] Oct 30 17:57:06.528596 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.528579 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.552263 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.552234 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9d75c56b-xtlgz"] Oct 30 17:57:06.592442 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592417 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592578 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592490 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592578 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592553 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592652 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592607 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592713 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592696 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592763 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592721 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592853 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592838 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.592908 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.592868 2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694185 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694159 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694185 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694187 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694598 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694216 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694598 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694236 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694598 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694302 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694598 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694328 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694598 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694361 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694598 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694533 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.694988 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694964 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.695167 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.694994 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.695272 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.695043 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.695272 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.695190 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.695272 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.695226 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.696686 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.696665 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.696850 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.696832 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.718761 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.718736 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:06.839676 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:06.839654 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:07.139679 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.139549 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-f9d75c56b-xtlgz"] Oct 30 17:57:07.142920 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:07.142897 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode23038e0_b513_49b0_bf52_f66a54d0c3d7.slice/crio-a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd WatchSource:0}: Error finding container a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd: Status 404 returned error can't find the container with id a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd Oct 30 17:57:07.193388 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.193357 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:07.416297 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.416220 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:07.459417 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.459395 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:07.464968 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.464938 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:07.473552 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.473525 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9d75c56b-xtlgz" event={"ID":"e23038e0-b513-49b0-bf52-f66a54d0c3d7","Type":"ContainerStarted","Data":"f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e"} Oct 30 17:57:07.473650 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.473566 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9d75c56b-xtlgz" event={"ID":"e23038e0-b513-49b0-bf52-f66a54d0c3d7","Type":"ContainerStarted","Data":"a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd"} Oct 30 17:57:07.556011 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.555967 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-f9d75c56b-xtlgz" podStartSLOduration=1.555955348 podStartE2EDuration="1.555955348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:07.555821443 +0000 UTC m=+62.002160833" watchObservedRunningTime="2026-10-30 17:57:07.555955348 +0000 UTC m=+62.002294736" Oct 30 17:57:07.675498 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:07.675416 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:08.767618 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:08.767583 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:08.978556 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:08.978518 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:09.443400 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:09.443362 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:09.650786 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:09.650755 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:09.856431 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:09.856396 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:10.064243 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:10.064208 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:11.850498 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.850440 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:11.850915 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.850521 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:11.864812 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.864787 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:57:11.873734 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.873710 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:11.877383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.877366 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:57:11.883408 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.883382 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:11.951733 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.951691 2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:11.966047 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.966027 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:57:11.986747 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.986725 2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:57:11.995201 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:11.995181 2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:12.063345 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.063324 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkbt2\"" Oct 30 17:57:12.069539 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.069523 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:12.074518 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.074505 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzlmk\"" Oct 30 17:57:12.080725 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.080708 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:12.081394 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.081375 2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dxtgs\"" Oct 30 17:57:12.087741 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.087723 2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:12.371360 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.371288 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-c8fkb"] Oct 30 17:57:12.374851 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:12.374826 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f41cfd0_1e7a_4972_9556_0a6b4ed7fc5b.slice/crio-f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3 WatchSource:0}: Error finding container f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3: Status 404 returned error can't find the container with id f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3 Oct 30 17:57:12.390415 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.390391 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-pjfgc"] Oct 30 17:57:12.393384 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:12.393357 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6990713_1f8a_4ecc_9279_9e28d2c0cba4.slice/crio-3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec WatchSource:0}: Error finding container 3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec: Status 404 returned error can't find the container with id 3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec Oct 30 17:57:12.404127 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.404069 2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqtbs"] Oct 30 17:57:12.407676 ip-10-0-136-241 kubenswrapper[2574]: W1030 17:57:12.407652 2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod788e0688_d9fb_48af_993d_c3bc6a0bc4c5.slice/crio-7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99 WatchSource:0}: Error finding container 7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99: Status 404 returned error can't find the container with id 7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99 Oct 30 17:57:12.493280 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.493253 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqtbs" event={"ID":"788e0688-d9fb-48af-993d-c3bc6a0bc4c5","Type":"ContainerStarted","Data":"7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99"} Oct 30 17:57:12.494418 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.494384 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c8fkb" event={"ID":"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b","Type":"ContainerStarted","Data":"f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3"} Oct 30 17:57:12.495579 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:12.495558 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjfgc" event={"ID":"a6990713-1f8a-4ecc-9279-9e28d2c0cba4","Type":"ContainerStarted","Data":"3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec"} Oct 30 17:57:13.502240 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:13.502204 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c8fkb" event={"ID":"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b","Type":"ContainerStarted","Data":"e278c914122f12b0580153b617471dcd47e2419644459e0b7a0002278c91d174"} Oct 30 17:57:14.033378 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:14.033342 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:14.508303 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:14.508260 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-c8fkb" event={"ID":"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b","Type":"ContainerStarted","Data":"b9c7c143dd23f18e53c9210fcf3437adde61d1f2cddc13167a07d3db7b856366"} Oct 30 17:57:14.769865 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:14.769775 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:14.974677 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:14.974642 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:16.272366 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.272330 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:16.499407 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.499366 2574 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:16.519224 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.519193 2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-pjfgc" event={"ID":"a6990713-1f8a-4ecc-9279-9e28d2c0cba4","Type":"ContainerStarted","Data":"a5fa014cc3ddbbb92c77e0c33400508053809ce24c40d01679f80b302fb8e008"} Oct 30 17:57:16.519383 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.519294 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:16.561485 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.561428 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-pjfgc" podStartSLOduration=68.294380372 podStartE2EDuration="1m11.561415s" totalImagesPullingTime="3.267034628s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:16.560393171 +0000 UTC m=+71.006732561" watchObservedRunningTime="2026-10-30 17:57:16.561415 +0000 UTC m=+71.007754438" Oct 30 17:57:16.561669 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.561646 2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-c8fkb" podStartSLOduration=70.592910804 podStartE2EDuration="1m11.561637253s" totalImagesPullingTime="968.726449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:56:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:14.5551367 +0000 UTC m=+69.001476092" watchObservedRunningTime="2026-10-30 17:57:16.561637253 +0000 UTC m=+71.007976642" Oct 30 17:57:16.840859 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.840787 2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:16.840859 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.840841 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:16.846534 ip-10-0-136-241 kubenswrapper[2574]: I1030 17:57:16.846503 2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:17.235521 ip-10-0-136-241 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 17:57:17.244944 ip-10-0-136-241 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 17:57:17.245107 ip-10-0-136-241 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 17:57:17.245152 ip-10-0-136-241 systemd[1]: kubelet.service: Consumed 4.604s CPU time. Oct 30 17:57:17.263070 ip-10-0-136-241 systemd[1]: Starting Kubernetes Kubelet... Oct 30 17:57:17.323839 ip-10-0-136-241 kubenswrapper[6966]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:17.323839 ip-10-0-136-241 kubenswrapper[6966]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 17:57:17.323839 ip-10-0-136-241 kubenswrapper[6966]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:17.323839 ip-10-0-136-241 kubenswrapper[6966]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 17:57:17.325851 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325780 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.325851 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325850 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.325851 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325854 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325857 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325860 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325863 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325866 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325868 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325872 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325874 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325877 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325880 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325882 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325885 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325895 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325898 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325901 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325903 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325906 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325909 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325911 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325914 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.325948 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325916 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325919 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325921 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325924 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325926 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325929 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325931 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325934 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325937 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325940 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325942 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325945 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325948 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325951 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325954 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325957 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325959 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325962 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325965 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.326438 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325968 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325971 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325973 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325976 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325978 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325981 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325984 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325987 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325990 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325992 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325995 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.325997 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326000 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326003 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326005 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326008 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326010 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326013 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326015 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.326956 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326018 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326021 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326023 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326026 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326029 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326032 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326034 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326037 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326040 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326043 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326045 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326049 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326053 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326056 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326061 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326064 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326068 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326071 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326074 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.327433 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326076 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326079 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326082 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326084 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326087 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326089 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326092 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326095 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326098 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326101 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326104 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326106 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326109 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326112 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326114 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326117 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326119 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326122 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326124 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.327943 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326127 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326130 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326133 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326136 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326139 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326141 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326144 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326146 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326940 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326949 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326952 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326955 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326958 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326961 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326963 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326967 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326970 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326973 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326976 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326979 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.328455 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326981 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326984 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326987 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326989 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326992 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326995 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.326997 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327000 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327003 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327005 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327008 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327010 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327015 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327018 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327020 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327023 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327026 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327028 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327031 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.328978 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327033 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327036 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327039 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327042 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327045 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327047 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327050 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327053 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327055 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327058 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327061 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327064 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327066 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327069 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327071 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327074 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327076 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327079 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327081 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327084 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.329448 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327086 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327089 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327091 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327094 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327096 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327100 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327102 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327105 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327108 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327110 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327113 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327115 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327118 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327120 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327123 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327126 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327129 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327131 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327134 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.329983 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327139 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327143 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327147 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327156 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327159 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327162 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327165 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327167 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327169 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327172 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327174 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327177 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327179 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327182 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327185 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327187 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327190 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327192 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327196 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327198 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.330471 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327201 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327204 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327206 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327210 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327214 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327217 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327220 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327226 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327229 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327232 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327235 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327238 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327240 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327243 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327245 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327248 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327751 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327757 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327760 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.330961 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327763 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327765 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327768 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327771 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327774 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327776 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327779 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327781 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327784 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327786 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327789 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327793 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327796 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327799 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327801 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327804 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327808 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327812 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327816 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.331567 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327818 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327822 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327825 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327828 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327830 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327833 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327835 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327838 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327840 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327843 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327846 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327849 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327852 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327854 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327857 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327859 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327862 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327864 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327867 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327869 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.332393 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327872 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327875 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327877 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327880 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327884 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327886 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327889 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327891 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327894 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327896 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327899 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327902 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327905 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327908 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327911 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327913 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327916 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327919 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327922 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327926 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.333287 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327929 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327932 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327935 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327939 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327942 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327945 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327948 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327951 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327953 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327956 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327959 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327961 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327964 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327966 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327969 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327971 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327975 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327977 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327980 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.334127 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327982 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327985 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327988 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327990 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327993 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327995 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.327999 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328002 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328004 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328007 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328009 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328012 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328016 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328018 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328021 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328023 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328026 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328029 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328031 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.334905 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328034 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328036 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328039 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328042 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328044 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328047 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328133 6966 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328140 6966 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328145 6966 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328150 6966 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328155 6966 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328158 6966 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328164 6966 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328168 6966 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328171 6966 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328175 6966 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328178 6966 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328181 6966 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328184 6966 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328189 6966 flags.go:64] FLAG: --cgroup-root="" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328192 6966 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328195 6966 flags.go:64] FLAG: --client-ca-file="" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328197 6966 flags.go:64] FLAG: --cloud-provider="external" Oct 30 17:57:17.335543 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328200 6966 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328204 6966 flags.go:64] FLAG: --cluster-domain="" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328207 6966 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328210 6966 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328213 6966 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328216 6966 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328220 6966 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328223 6966 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328227 6966 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328230 6966 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328233 6966 flags.go:64] FLAG: --contention-profiling="false" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328236 6966 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328239 6966 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328242 6966 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328245 6966 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328249 6966 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328252 6966 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328255 6966 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328258 6966 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328261 6966 flags.go:64] FLAG: --enable-server="true" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328264 6966 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328268 6966 flags.go:64] FLAG: --event-burst="100" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328271 6966 flags.go:64] FLAG: --event-qps="50" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328274 6966 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 17:57:17.336095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328278 6966 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328281 6966 flags.go:64] FLAG: --eviction-hard="" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328285 6966 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328288 6966 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328292 6966 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328295 6966 flags.go:64] FLAG: --eviction-soft="" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328298 6966 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328301 6966 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328304 6966 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328307 6966 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328310 6966 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328313 6966 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328316 6966 flags.go:64] FLAG: --feature-gates="" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328320 6966 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328323 6966 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328326 6966 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328329 6966 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328332 6966 flags.go:64] FLAG: --healthz-port="10248" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328336 6966 flags.go:64] FLAG: --help="false" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328338 6966 flags.go:64] FLAG: --hostname-override="ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328342 6966 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328345 6966 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328348 6966 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328351 6966 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 17:57:17.336720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328355 6966 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328358 6966 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328361 6966 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328363 6966 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328366 6966 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328369 6966 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328372 6966 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328375 6966 flags.go:64] FLAG: --kube-reserved="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328378 6966 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328381 6966 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328384 6966 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328387 6966 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328391 6966 flags.go:64] FLAG: --lock-file="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328394 6966 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328397 6966 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328400 6966 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328406 6966 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328408 6966 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328412 6966 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328415 6966 flags.go:64] FLAG: --logging-format="text" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328418 6966 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328421 6966 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328424 6966 flags.go:64] FLAG: --manifest-url="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328427 6966 flags.go:64] FLAG: --manifest-url-header="" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328431 6966 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 17:57:17.337307 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328434 6966 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328439 6966 flags.go:64] FLAG: --max-pods="110" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328443 6966 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328446 6966 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328450 6966 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328454 6966 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328469 6966 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328472 6966 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328476 6966 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328482 6966 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328485 6966 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328488 6966 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328491 6966 flags.go:64] FLAG: --pod-cidr="" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328494 6966 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328497 6966 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328500 6966 flags.go:64] FLAG: --pods-per-core="0" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328503 6966 flags.go:64] FLAG: --port="10250" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328506 6966 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328509 6966 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0584a2fed4f43192f" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328513 6966 flags.go:64] FLAG: --qos-reserved="" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328516 6966 flags.go:64] FLAG: --read-only-port="10255" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328519 6966 flags.go:64] FLAG: --register-node="true" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328522 6966 flags.go:64] FLAG: --register-with-taints="" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328525 6966 flags.go:64] FLAG: --registry-burst="10" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328528 6966 flags.go:64] FLAG: --registry-qps="5" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328531 6966 flags.go:64] FLAG: --reserved-cpus="" Oct 30 17:57:17.337940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328534 6966 flags.go:64] FLAG: --reserved-memory="" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328538 6966 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328541 6966 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328544 6966 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328547 6966 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328549 6966 flags.go:64] FLAG: --runonce="false" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328552 6966 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328555 6966 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328559 6966 flags.go:64] FLAG: --seccomp-default="false" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328562 6966 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328566 6966 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328569 6966 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328572 6966 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328576 6966 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328579 6966 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328582 6966 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328584 6966 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328587 6966 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328590 6966 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328593 6966 flags.go:64] FLAG: --system-cgroups="" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328596 6966 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328601 6966 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328604 6966 flags.go:64] FLAG: --tls-cert-file="" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328607 6966 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328611 6966 flags.go:64] FLAG: --tls-min-version="" Oct 30 17:57:17.338712 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328613 6966 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328616 6966 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328619 6966 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328622 6966 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328625 6966 flags.go:64] FLAG: --v="2" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328629 6966 flags.go:64] FLAG: --version="false" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328633 6966 flags.go:64] FLAG: --vmodule="" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328636 6966 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.328639 6966 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328766 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328770 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328773 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328776 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328778 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328781 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328784 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328787 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328791 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328793 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328796 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328798 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.339321 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328801 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328804 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328806 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328809 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328811 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328815 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328819 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328822 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328824 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328827 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328830 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328832 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328835 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328838 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328840 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328843 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328845 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328848 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328850 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328853 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.340107 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328856 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328858 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328861 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328863 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328866 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328869 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328872 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328874 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328879 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328881 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328884 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328886 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328889 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328891 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328894 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328896 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328899 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328901 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328904 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.340859 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328908 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328912 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328915 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328918 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328921 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328923 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328926 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328929 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328931 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328934 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328936 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328940 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328943 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328945 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328948 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328950 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328953 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328956 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328958 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.341389 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328961 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328963 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328967 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328970 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328973 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328976 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328979 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328982 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328984 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328987 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328990 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328993 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328995 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.328998 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329001 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329003 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329006 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329008 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329010 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329013 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.341887 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329016 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329018 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329020 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329023 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329027 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329029 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329032 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329034 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329037 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329040 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329042 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329045 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329047 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329050 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329054 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329056 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.342380 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.329081 6966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329114 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329118 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329121 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329124 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329126 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329129 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329133 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329135 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329138 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329141 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329144 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329146 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329149 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329151 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329154 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329157 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329159 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329162 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.342910 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329164 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329167 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329172 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329175 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329178 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329181 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329183 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329186 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329188 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329191 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329194 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329197 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329200 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329202 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329205 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329208 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329210 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329213 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329216 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329218 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.343480 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329220 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329224 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329226 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329229 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329231 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329234 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329236 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329239 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329241 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329244 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329246 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329249 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329251 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329254 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329258 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329260 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329264 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329267 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329270 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.343998 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329273 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329276 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329279 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329281 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329285 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329288 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329290 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329294 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329297 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329299 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329302 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329304 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329307 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329310 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329312 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329315 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329317 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329320 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329322 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.344478 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329325 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329327 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329330 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329332 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329335 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329337 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329341 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329343 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329347 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329350 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329353 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329356 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329358 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329361 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329363 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329366 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329369 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329372 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329375 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329377 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.344930 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329380 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329383 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329386 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329388 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329391 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329393 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329396 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329398 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329401 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.329403 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.335420 6966 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.335437 6966 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.335454 6966 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.335479 6966 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335550 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335555 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335558 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335561 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335564 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335567 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.345437 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335570 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335573 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335575 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335578 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335581 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335584 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335587 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335589 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335592 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335595 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335597 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335600 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335603 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335606 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335610 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335614 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335618 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335621 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335624 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335627 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.345997 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335629 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335633 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335636 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335639 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335641 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335644 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335647 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335650 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335652 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335655 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335657 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335660 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335663 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335665 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335668 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335671 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335673 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335676 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335679 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.346502 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335683 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335689 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335692 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335695 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335699 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335701 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335704 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335707 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335710 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335713 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335715 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335718 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335721 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335723 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335726 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335729 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335732 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335734 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335737 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.347056 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335740 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335742 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335745 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335748 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335750 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335753 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335756 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335759 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335762 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335764 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335767 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335770 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335773 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335777 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335779 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335782 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335785 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335788 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335791 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.347528 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335794 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335796 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335799 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335801 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335804 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335806 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335809 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335811 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335814 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335817 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335819 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335822 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335824 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335827 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335830 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335832 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335835 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335838 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335840 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335843 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.348130 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335845 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335848 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.335850 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.335878 6966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336033 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336038 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336041 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336044 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336047 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336050 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336053 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336056 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336058 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336061 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336064 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.348654 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336066 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336069 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336071 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336074 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336077 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336080 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336082 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336085 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336087 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336090 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336092 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336095 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336097 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336100 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336103 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336106 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336109 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336111 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336114 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336117 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.349025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336119 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336122 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336124 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336127 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336130 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336132 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336135 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336138 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336141 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336144 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336147 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336149 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336151 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336154 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336156 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336159 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336161 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336164 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336167 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336169 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.349605 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336172 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336174 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336177 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336180 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336182 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336185 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336187 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336190 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336192 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336195 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336197 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336201 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336205 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336208 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336212 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336215 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336217 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336220 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336222 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336225 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.350410 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336228 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336231 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336233 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336236 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336238 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336242 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336245 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336248 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336251 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336254 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336256 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336259 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336261 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336264 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336266 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336269 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336272 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336274 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336277 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.351012 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336279 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336282 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336284 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336287 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336290 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336292 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336295 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336297 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336300 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336302 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336305 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336307 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336310 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336312 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336318 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.351576 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336320 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.336343 6966 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336380 6966 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336384 6966 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336387 6966 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336390 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336393 6966 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336395 6966 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336398 6966 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336401 6966 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336403 6966 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336406 6966 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336409 6966 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336411 6966 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336414 6966 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336417 6966 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 17:57:17.351939 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336419 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336422 6966 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336425 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336428 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336431 6966 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336433 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336436 6966 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336439 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336441 6966 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336444 6966 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336447 6966 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336450 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336452 6966 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336455 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336472 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336476 6966 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336482 6966 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336485 6966 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336487 6966 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336490 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 17:57:17.352338 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336493 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336495 6966 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336498 6966 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336501 6966 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336504 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336507 6966 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336510 6966 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336512 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336515 6966 feature_gate.go:386] unrecognized feature gate: Example Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336518 6966 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336520 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336523 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336527 6966 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336531 6966 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336534 6966 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336538 6966 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336540 6966 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336543 6966 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336545 6966 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336548 6966 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 17:57:17.352837 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336551 6966 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336553 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336555 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336558 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336561 6966 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336563 6966 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336567 6966 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336571 6966 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336575 6966 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336579 6966 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336582 6966 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336584 6966 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336587 6966 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336589 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336592 6966 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336595 6966 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336598 6966 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336600 6966 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336603 6966 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 17:57:17.353359 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336606 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336608 6966 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336611 6966 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336613 6966 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336616 6966 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336619 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336621 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336624 6966 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336626 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336629 6966 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336632 6966 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336634 6966 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336637 6966 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336639 6966 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336642 6966 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336644 6966 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336647 6966 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336649 6966 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336652 6966 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 17:57:17.353855 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336654 6966 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336657 6966 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336661 6966 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336664 6966 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336667 6966 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336670 6966 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336673 6966 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336675 6966 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336678 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336680 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336683 6966 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336686 6966 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336689 6966 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:17.336692 6966 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.336746 6966 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.337689 6966 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.337741 6966 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.337950 6966 server.go:1014] "Starting client certificate rotation" Oct 30 17:57:17.354325 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.338106 6966 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.338175 6966 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:51:05 +0000 UTC" deadline="2028-04-16 08:29:39.794666487 +0000 UTC" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.338214 6966 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12806h32m22.456455408s" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.338370 6966 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.339425 6966 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.340688 6966 log.go:25] "Validated CRI v1 runtime API" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.342503 6966 log.go:25] "Validated CRI v1 image API" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.343236 6966 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 17:57:17.354819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.345326 6966 fs.go:135] Filesystem UUIDs: map[3eb9a2ff-93e1-466f-87b8-1b6c46daa136:/dev/nvme0n1p4 48028ed0-b1aa-4235-9874-b0471ad1d03a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 17:57:17.355051 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.345345 6966 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629/userdata/shm major:0 minor:355 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5/userdata/shm major:0 minor:299 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec/userdata/shm major:0 minor:470 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/731a4a90ddf12d02def07224fa394c694121fc7436bfbadaa2ef9ef5e3312cc5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/731a4a90ddf12d02def07224fa394c694121fc7436bfbadaa2ef9ef5e3312cc5/userdata/shm major:0 minor:259 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e/userdata/shm major:0 minor:442 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99/userdata/shm major:0 minor:472 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/89a4d82ffb50b918a668728c562d5ccc314756d8025be54a609c96c44f548f3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/89a4d82ffb50b918a668728c562d5ccc314756d8025be54a609c96c44f548f3c/userdata/shm major:0 minor:347 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd/userdata/shm major:0 minor:455 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef/userdata/shm major:0 minor:395 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4/userdata/shm major:0 minor:252 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e7a9612b393a7e97752ef9e32e95c167166ead35445eb6f1636e84ec8987a386/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e7a9612b393a7e97752ef9e32e95c167166ead35445eb6f1636e84ec8987a386/userdata/shm major:0 minor:258 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3/userdata/shm major:0 minor:468 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~projected/kube-api-access-qwnl8:{mountpoint:/var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~projected/kube-api-access-qwnl8 major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b/volumes/kubernetes.io~projected/kube-api-access-l9s6l:{mountpoint:/var/lib/kubelet/pods/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b/volumes/kubernetes.io~projected/kube-api-access-l9s6l major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b/volumes/kubernetes.io~secret/metrics-certs major:0 minor:461 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/447f1a32-218e-4599-9a99-011281767a95/volumes/kubernetes.io~projected/kube-api-access-wtg4z:{mountpoint:/var/lib/kubelet/pods/447f1a32-218e-4599-9a99-011281767a95/volumes/kubernetes.io~projected/kube-api-access-wtg4z major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/447f1a32-218e-4599-9a99-011281767a95/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/447f1a32-218e-4599-9a99-011281767a95/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~projected/kube-api-access-khtd2:{mountpoint:/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~projected/kube-api-access-khtd2 major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~empty-dir/tmp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~projected/kube-api-access-s9jjx:{mountpoint:/var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~projected/kube-api-access-s9jjx major:0 minor:69 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5b91437e-1659-46b1-bc95-687ddd06c392/volumes/kubernetes.io~projected/kube-api-access-qgg2b:{mountpoint:/var/lib/kubelet/pods/5b91437e-1659-46b1-bc95-687ddd06c392/volumes/kubernetes.io~projected/kube-api-access-qgg2b major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/64fed540-f4cb-4e19-b3d9-7531e486c4f5/volumes/kubernetes.io~projected/kube-api-access-6f5xs:{mountpoint:/var/lib/kubelet/pods/64fed540-f4cb-4e19-b3d9-7531e486c4f5/volumes/kubernetes.io~projected/kube-api-access-6f5xs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/699b6cb3-446f-440e-8d68-66c4e09ee322/volumes/kubernetes.io~projected/kube-api-access-v8rqz:{mountpoint:/var/lib/kubelet/pods/699b6cb3-446f-440e-8d68-66c4e09ee322/volumes/kubernetes.io~projected/kube-api-access-v8rqz major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/699b6cb3-446f-440e-8d68-66c4e09ee322/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/699b6cb3-446f-440e-8d68-66c4e09ee322/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/788e0688-d9fb-48af-993d-c3bc6a0bc4c5/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/788e0688-d9fb-48af-993d-c3bc6a0bc4c5/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:466 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/volumes/kubernetes.io~projected/kube-api-access-m8mhq:{mountpoint:/var/lib/kubelet/pods/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/volumes/kubernetes.io~projected/kube-api-access-m8mhq major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~projected/kube-api-access-g9szl:{mountpoint:/var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~projected/kube-api-access-g9szl major:0 minor:251 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:247 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:248 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cc87edf-5705-49bc-abe7-7cf921ed75ab/volumes/kubernetes.io~projected/kube-api-access-h6pm8:{mountpoint:/var/lib/kubelet/pods/9cc87edf-5705-49bc-abe7-7cf921ed75ab/volumes/kubernetes.io~projected/kube-api-access-h6pm8 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4/volumes/kubernetes.io~projected/kube-api-access-z8tjv:{mountpoint:/var/lib/kubelet/pods/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4/volumes/kubernetes.io~projected/kube-api-access-z8tjv major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volumes/kubernetes.io~projected/kube-api-access-xlpnz:{mountpoint:/var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volumes/kubernetes.io~projected/kube-api-access-xlpnz major:0 minor:68 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~projected/kube-api-access-zdw5k:{mountpoint:/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~projected/kube-api-access-zdw5k major:0 minor:441 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:440 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:435 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6990713-1f8a-4ecc-9279-9e28d2c0cba4/volumes/kubernetes.io~projected/kube-api-access-gp5mb:{mountpoint:/var/lib/kubelet/pods/a6990713-1f8a-4ecc-9279-9e28d2c0cba4/volumes/kubernetes.io~projected/kube-api-access-gp5mb major:0 minor:467 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/alertmanager/9 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/config-reloader/6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~empty-dir/config-out major:0 minor:297 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~projected/kube-api-access-56kz9:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~projected/kube-api-access-56kz9 major:0 minor:298 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~projected/tls-assets major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:296 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/config-volume major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/web-config major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bbd7a723-acf0-4c7c-95a5-2bd54d65460d/volumes/kubernetes.io~projected/kube-api-access-tbst6:{mountpoint:/var/lib/kubelet/pods/bbd7a723-acf0-4c7c-95a5-2bd54d65460d/volumes/kubernetes.io~projected/kube-api-access-tbst6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc0cafd7-e383-4391-8d85-073289c3f442/volumes/kubernetes.io~projected/kube-api-access-7q4kz:{mountpoint:/var/lib/kubelet/pods/bc0cafd7-e383-4391-8d85-073289c3f442/volumes/kubernetes.io~projected/kube-api-access-7q4kz major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc0cafd7-e383-4391-8d85-073289c3f442/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/bc0cafd7-e383-4391-8d85-073289c3f442/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~projected/kube-api-access-4r5s8:{mountpoint:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~projected/kube-api-access-4r5s8 major:0 minor:354 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:351 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:352 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:353 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cfcad2e0-540b-4e42-ba77-303e060d37bd/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cfcad2e0-540b-4e42-ba77-303e060d37bd/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~projected/kube-api-access-wfv4g:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~projected/kube-api-access-wfv4g major:0 minor:346 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:344 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:345 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:337 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:341 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~projected/kube-api-access-cfq5h:{mountpoint:/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~projected/kube-api-access-cfq5h major:0 minor:454 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:448 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:453 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~projected/kube-api-access-vp6xh:{mountpoint:/var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~projected/kube-api-access-vp6xh major:0 minor:250 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:249 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4303592-c23b-49bc-a242-dfefafe43d0a/volumes/kubernetes.io~projected/kube-api-access-kxbpv:{mountpoint:/var/lib/kubelet/pods/f4303592-c23b-49bc-a242-dfefafe43d0a/volumes/kubernetes.io~projected/kube-api-access-kxbpv major:0 minor:64 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/24d405cb61f820f6630c99481a34cc0d1287b878cfc8109efbb77e2761503c57/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/d6e7d5f4fa905326f434806f861aa0498758d8cd1f5f23ab94815928e1388ace/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/4641b5a4fc82d127017402ddbaf12e822b2774f3e5a446e0365faf8370ff7d90/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/087befc1b3055e25eb527b323651420000a2723a1771bbb5fb9a1634a809042b/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/b71156837025bd36f70692f8440dd345f19f8023b257b113c904c492762a4428/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/584ea9a9633ce4696d28646d1e8bd4dc2899605973ae89bac9fee54a081d3e86/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c8111c86abb1caa1685dc16a5464693d647a923efe6568dcc5a4b115183deeac/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/e151ab53a03ea5de3b89a9478e26ff371ad563212835504b3aed0bff3c90d130/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/2c69af345510dc1e30f0736fb3d603c0c2bd93846700f5323cc35aa40483fea9/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/6aba115ba49f1a7579fa46c983a439ea55ffef189a2422690a48523ae13ad433/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/518ca79af5c49f5f2ed14c9a845c72fc15b995970eb0a1a1b50aae6970507e3f/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/6c9bfa0f51bff43dc98d25051ffb9519a92bfc105ad9445e4cc3f68bfce5d19e/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d2afc63efa39c4e42e31987544c80770ccb841fb20b47204d101d5da816381ab/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/d8dd097cf7e84e5eff0f8b90686e100fa918958a1a8ee165c2d4ddeaffbf1e82/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9327065eb2ab74b9194a1863a5a4cef2aa434cb2a7b69dd2107ea2076787d12a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/8847197b9d57f12aeb9dd0d9fdcbae52868405d0e92bfe076a856099e7651ae1/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/00df592919e0de0c500d6c4b058c5385f12e9cda6b62ffd93017705e19ad7ecd/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/dd9e4e5cdf6667330e2c02eff7d30e41f8a091c9b4d833cef218f22844d2aa3f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/13ece3e397de8e57eb7a0d83e65873c51093b3f3b8c44465d4cac0d3d1e35b2a/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/e4bbf6eb206430dd423442aead77fbc7f36322962c3ae0b9c0961900db579d49/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/c3c4af102ef25515f2b5dc70bb82186c6a2aabebb26fd6cab284e60bbe701624/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/324a6d2734a0245c1b743c73bc5c745ca04461c2b67ffb8ba731ef9de1fd7a6c/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/326c8643847bba4dfe98133783f2fb79cbb0babebb1866039022d78e39535a8d/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/327bd89d67a6038f309aa38dbe168b52a26713678f0a9085becc7a4e8b427b5f/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/4a9013e5cfefa43d22e5e0e4e81f8faefc98d41401b18b05d21c19446ad0b7bb/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/333a78fe774e0e88d060ec0ee73ec2c9ff29a9d1d64c4c1bf27738f73c28559a/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-232:{mountpoint:/var/lib/containers/storage/overlay/050007cdb25cbe6e0329259a099101562f3f94c1bbfd029da355015e6b9adaff/merged major:0 minor:232 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/b296b10162b184a6a259c5ba7f125cc19e293ce9ceb74b0b034f5f9f39d9ce95/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/dc480d84390490f8b00726b2cce713943824313523baf6d3703071b346ed182c/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-263:{mountpoint:/var/lib/containers/storage/overlay/f238bb6ad5eeb72df5c83b23e65810ec5aad77299527f2c06b9777f6b0a1d193/merged major:0 minor:263 fsType:overlay blockSize:0} overlay_0-265:{mountpoint:/var/lib/containers/storage/overlay/6b16b9b9cc8b71ae2d7adea3ff21c0c764cc9c120b8afe19fb44734dce829502/merged major:0 minor:265 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/e073cd67d2ddb670a9e9ae3c6945127493f788a588923c425ba2bcc1081c73a2/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/d3e963b02161302923cd7e493357f7ae784338432f285ebb0542d49c793e75ed/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-279:{mountpoint:/var/lib/containers/storage/overlay/100d2c14c92e455e2ecedf7195317c54766b86ce6c021fc4eecbb64897850530/merged major:0 minor:279 fsType:overlay blockSize:0} overlay_0-285:{mountpoint:/var/lib/containers/storage/overlay/fbfd02c2b73832cbd83dcc23d50dd9d3b630e39e4358ed4d16263a74b3ad442b/merged major:0 minor:285 fsType:overlay blockSize:0} overlay_0-301:{mountpoint:/var/lib/containers/storage/overlay/ecfd870d3334a9ef62eabe9110288bac1f570a312301185aa5c7a5aa5f5c370e/merged major:0 minor:301 fsType:overlay blockSize:0} overlay_0-303:{mountpoint:/var/lib/containers/storage/overlay/39238aefcc60690600b02d154cfe5761bb8b194813be9f31f3769cb9be1e94f6/merged major:0 minor:303 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/4e7727a2ae88a0ad85701fffdb7ce8361f8c9a142536826067fb4be0da99cdcc/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/6e08adc76a20a24645fdd09ad89a8112b68eb0c7be2ab3c6bcd5d2b6a7455e1f/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/5bb7afa9075b2a67ab2f9b0fb784be41594f3f2f92573b4ca15c6c43b2f60e88/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/7fcd8608d6adef9cbbf845aa84e2df5d338cae30eac73563e5ede5b6ce18cbde/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/b75be77759cf174e424cb4bdca86c05544716d65a2387b32045ae0ea44624e37/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/b326a852433bd3ee4cf6dd5bb5cfb77086f2a1fbc8e8eb72011e7148203bd02f/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/1bcdf620cd68855351b6e635755b27d4d07ee33468ef4c27ea21f0bc239fc530/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-357:{mountpoint:/var/lib/containers/storage/overlay/c36ad3469ec2e1c72c00f7806e03e42c2a30a1d39e6f45850a69e54644c88f09/merged major:0 minor:357 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/d3f347f799faedd48d28282cbf7bb653b42fb91359efca8327070ebaa73496de/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-365:{mountpoint:/var/lib/containers/storage/overlay/0a2be930c9cf27417346ebaa839aec7478a11dd3172c7db840ff568fbf854595/merged major:0 minor:365 fsType:overlay blockSize:0} overlay_0-374:{mountpoint:/var/lib/containers/storage/overlay/0a4ad66dbf6c4ac97af26797a44107fe7519f723d661d70117a6280e9d902ae8/merged major:0 minor:374 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/02eb859b209552585a0c2df2c4a248f50bd9948c783aaee9ee79610a05cec226/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/77f4ffd88b056ee240511a22897df2df42229d016507f6fa64426e4e624b0f84/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-387:{mountpoint:/var/lib/containers/storage/overlay/28bf4c42b9e338b44b094b3e1b25743d2c4d3bbbd45e268a20c4f3d2339826b4/merged major:0 minor:387 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/59aacba20b4b9166c848231d4f361e740f30b2a8cc0cd9849bf966de06552f16/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-400:{mountpoint:/var/lib/containers/storage/overlay/5f80c32f6df3c02aed97e4753a8f5e2a56bee92f17238b49f905a605a671a053/merged major:0 minor:400 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/673288f7d1ceb15df1a86e202b7b3363ddc9177916548b30101e34bdc3ada5e9/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/b81b8b1d4485e6980601ae627222c5d69025060f67dbe552e17c3d1e005b94fb/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/68be8450094349a958695f81254d53098508d6bfa724a9447e9f8fcc9edc8d00/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/deb51a6565db5c187538dfabb4850acdbef928f80ee0c2d50864cd581cc32f83/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/bee22a1abab8bd4aeceb419b59e6280b17629a58df619b46daa268539043f0a8/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-421:{mountpoint:/var/lib/containers/storage/overlay/1975a4f79a3ec03db7030f101e8cc2cd990f038cb7784dbb00e459a0e08f5de8/merged major:0 minor:421 fsType:overlay blockSize:0} overlay_0-429:{mountpoint:/var/lib/containers/storage/overlay/58be6b4c05b9ba944e4ecc05a0b5518c1830a497f6a727ef97e4a9db17d458ea/merged major:0 minor:429 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/24d38be590e27b96f86642704c7875d838e531be3eb1f387307fa4b9cf19466d/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/ed3e5b5d9a0858dbb8c020a2117f4fb10aebfb50adc3d92a155b6dd7016af2e0/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-457:{mountpoint:/var/lib/containers/storage/overlay/085ce61a9cce37a4361ca786bcc1e63b0b0a354f8c636888aaac5887c597b7dc/merged major:0 minor:457 fsType:overlay blockSize:0} overlay_0-459:{mountpoint:/var/lib/containers/storage/overlay/de239e5c2878d94e0427f30295c05eab0216198ea852982f70e322dbea7bfa44/merged major:0 minor:459 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/2f359d29a17ab6835a26fe84b5278badeeba996fbdfb2dae1792eacadbe9c6ad/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-476:{mountpoint:/var/lib/containers/storage/overlay/9c02ef9ecdea2865eb511060ec10ae3837b5269e902850fe3feadaaf8b4d6f8a/merged major:0 minor:476 fsType:overlay blockSize:0} overlay_0-478:{mountpoint:/var/lib/containers/storage/overlay/e6407a2978b5b305cdc8eaf74f302032655beeb62874a05f6764f8f2a528694d/merged major:0 minor:478 fsType:overlay blockSize:0} overlay_0-480:{mountpoint:/var/lib/containers/storage/overlay/c4dea202e0df3ab46841407f84d5f4583c8e9aa8501327baeea3bbbc8f3fb7d1/merged major:0 minor:480 fsType:overlay blockSize:0} overlay_0-482:{mountpoint:/var/lib/containers/storage/overlay/f271635c14bc01bf46d14b1a70258ea3b590f7188290246215efd43d4b23a80e/merged major:0 minor:482 fsType:overlay blockSize:0} overlay_0-488:{mountpoint:/var/lib/containers/storage/overlay/a4292bad396e1a1bd5804f8f1a3f968bc517a9baecdb98c1b10e2fbec9d932ce/merged major:0 minor:488 fsType:overlay blockSize:0} overlay_0-493:{mountpoint:/var/lib/containers/storage/overlay/064f4f5e6d6c904d69671b7a2b7f1549bf9837dc4820612d63482e9f01188766/merged major:0 minor:493 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/ad5c7617d363d7cd2434bf96cbf1fdfad75533806b7702dc275c9ab908c177af/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/62279f847b757d1290d3ca5c98d52f656d7539e478e894e686b79c79ddec065e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-60:{mountpoint:/var/lib/containers/storage/overlay/b6bf56ea2a41a132b7449b3c6b7b0c875c061330307f6f777aecda37e07fa856/merged major:0 minor:60 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/f1629c6f86106284936bf290ebd5945aa34bcc3b29c211d309f5c5ffb22c4343/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/fbb6a325996467e59f2aeb0a4d159a55db31411c0cb0eac7f1655b25e62f8032/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/e7c26d87615e405b8bfd60ef5cd37b070e2ee0246781c50e753aad31910971a2/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/6d76b1688c47b449a941cfe915aa01c965c84845578204209887a6cd50781245/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/664a2ccd87012c5505842303ce8f5968cd96b554e58d5207a376469e6b39b0ad/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4b094f862e48afa60d37831c69ee3368e75437f369faed96f6e661eb2eea7d64/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/c6738a38a91e7e574976897982097a983285949a5ddcdd787603d342f79ed548/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/ed830258151bde5107e3a8468ea2bfa65e780b6e78257fe1c6fce9816140a74b/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.353734 6966 manager.go:217] Machine: {Timestamp:2026-10-30 17:57:17.353244518 +0000 UTC m=+0.059429865 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3119392 MemoryCapacity:33164328960 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c0abca374b8aaca9fdc6fb6272d1f SystemUUID:ec2c0abc-a374-b8aa-ca9f-dc6fb6272d1f BootID:97be8be2-c13d-43ec-8759-bbafc2aa8a46 Filesystems:[{Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-429 DeviceMajor:0 DeviceMinor:429 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e3baf4bccdfd56fbed0bf2335b18fa8b6ccab3df7eb58b9b3e00e57a53178ec/userdata/shm DeviceMajor:0 DeviceMinor:470 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b/volumes/kubernetes.io~projected/kube-api-access-l9s6l DeviceMajor:0 DeviceMinor:65 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6990713-1f8a-4ecc-9279-9e28d2c0cba4/volumes/kubernetes.io~projected/kube-api-access-gp5mb DeviceMajor:0 DeviceMinor:467 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~projected/kube-api-access-vp6xh DeviceMajor:0 DeviceMinor:250 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-265 DeviceMajor:0 DeviceMinor:265 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-357 DeviceMajor:0 DeviceMinor:357 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:287 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~projected/kube-api-access-s9jjx DeviceMajor:0 DeviceMinor:69 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc0cafd7-e383-4391-8d85-073289c3f442/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cc03cc009a1e0fb7d0ba07c0c20b1273859dbaa9510c9b6ab20c91e5418d78f4/userdata/shm DeviceMajor:0 DeviceMinor:252 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6c1b73ed3b132a9d2d5a28a8b08280f9f3b5c8efa290a1488bd788921b9c22f5/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-365 DeviceMajor:0 DeviceMinor:365 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:353 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~projected/kube-api-access-cfq5h DeviceMajor:0 DeviceMinor:454 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-478 DeviceMajor:0 DeviceMinor:478 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/80547547d8ddab426dd8cf6f55a24cf02bd38edf650b4dfcab311ac89a9a635c/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-279 DeviceMajor:0 DeviceMinor:279 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:288 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-493 DeviceMajor:0 DeviceMinor:493 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/699b6cb3-446f-440e-8d68-66c4e09ee322/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~projected/kube-api-access-khtd2 DeviceMajor:0 DeviceMinor:213 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/89a4d82ffb50b918a668728c562d5ccc314756d8025be54a609c96c44f548f3c/userdata/shm DeviceMajor:0 DeviceMinor:347 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:296 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-482 DeviceMajor:0 DeviceMinor:482 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:237 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-285 DeviceMajor:0 DeviceMinor:285 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:343 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-374 DeviceMajor:0 DeviceMinor:374 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/994dfd7f3fb39976f4ffe3e2385c8f4338513cb79a1e9941900be7da2b406e19/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/598d4aa001176478b5c0ae4e6a2f07630b8ac27e603dcce5d2deea37768abbc3/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:212 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:440 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~projected/kube-api-access-zdw5k DeviceMajor:0 DeviceMinor:441 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/f4303592-c23b-49bc-a242-dfefafe43d0a/volumes/kubernetes.io~projected/kube-api-access-kxbpv DeviceMajor:0 DeviceMinor:64 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:284 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5/userdata/shm DeviceMajor:0 DeviceMinor:299 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-301 DeviceMajor:0 DeviceMinor:301 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/01805ad6a87c74b0e5b6d31b1d570f43324269df963a3c88cdd4ff0938469629/userdata/shm DeviceMajor:0 DeviceMinor:355 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7dac5eb65ad2969a95f562cdc5e586429d84e712702f9381fece04ddfb04cb99/userdata/shm DeviceMajor:0 DeviceMinor:472 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9cc87edf-5705-49bc-abe7-7cf921ed75ab/volumes/kubernetes.io~projected/kube-api-access-h6pm8 DeviceMajor:0 DeviceMinor:66 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volumes/kubernetes.io~projected/kube-api-access-xlpnz DeviceMajor:0 DeviceMinor:68 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/volumes/kubernetes.io~projected/kube-api-access-m8mhq DeviceMajor:0 DeviceMinor:204 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:287 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:448 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-457 DeviceMajor:0 DeviceMinor:457 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:291 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-421 DeviceMajor:0 DeviceMinor:421 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc0cafd7-e383-4391-8d85-073289c3f442/volumes/kubernetes.io~projected/kube-api-access-7q4kz DeviceMajor:0 DeviceMinor:202 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/75c8b94a87abd92678e048ac59b916a18defa252250aa0575261385146ed7ffc/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:238 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-303 DeviceMajor:0 DeviceMinor:303 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a94d6d3495d521407a98c8694d98be0f040f224a308a085d04cce1812f201fef/userdata/shm DeviceMajor:0 DeviceMinor:395 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-480 DeviceMajor:0 DeviceMinor:480 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:59 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:287 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd/userdata/shm DeviceMajor:0 DeviceMinor:455 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d111489-3f0f-49bb-9de2-f400960f01d7/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e7a9612b393a7e97752ef9e32e95c167166ead35445eb6f1636e84ec8987a386/userdata/shm DeviceMajor:0 DeviceMinor:258 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-400 DeviceMajor:0 DeviceMinor:400 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cfcad2e0-540b-4e42-ba77-303e060d37bd/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~projected/kube-api-access-g9szl DeviceMajor:0 DeviceMinor:251 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~projected/kube-api-access-56kz9 DeviceMajor:0 DeviceMinor:298 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ef154e61aff3cc8d7aacf206ddeea5d0e9b203b827f9af80d1ad722908910932/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:248 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:337 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:287 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/447f1a32-218e-4599-9a99-011281767a95/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:381 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:290 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:297 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e68154455a8da6fcf85478e879d0207673b533ec84247f410b861467c3d9f92c/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/699b6cb3-446f-440e-8d68-66c4e09ee322/volumes/kubernetes.io~projected/kube-api-access-v8rqz DeviceMajor:0 DeviceMinor:203 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-263 DeviceMajor:0 DeviceMinor:263 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-459 DeviceMajor:0 DeviceMinor:459 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-476 DeviceMajor:0 DeviceMinor:476 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bbd7a723-acf0-4c7c-95a5-2bd54d65460d/volumes/kubernetes.io~projected/kube-api-access-tbst6 DeviceMajor:0 DeviceMinor:67 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0d41196f5d8f7974f920528f9e8992a3bce7a2fcd75a023da5b28f2988030f33/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:278 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:344 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:351 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:461 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5b91437e-1659-46b1-bc95-687ddd06c392/volumes/kubernetes.io~projected/kube-api-access-qgg2b DeviceMajor:0 DeviceMinor:63 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:211 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~projected/kube-api-access-wfv4g DeviceMajor:0 DeviceMinor:346 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:435 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:249 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/731a4a90ddf12d02def07224fa394c694121fc7436bfbadaa2ef9ef5e3312cc5/userdata/shm DeviceMajor:0 DeviceMinor:259 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~projected/kube-api-access-4r5s8 DeviceMajor:0 DeviceMinor:354 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e/userdata/shm DeviceMajor:0 DeviceMinor:442 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4/volumes/kubernetes.io~projected/kube-api-access-z8tjv DeviceMajor:0 DeviceMinor:62 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e05273d4a0658624ddd8fbe374d05bafbdccdbb9a0893973b00730204a39cea5/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:342 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:341 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:453 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6917aa782e0c79ea7a0343d80345bdf8db20e1c8f7b5e7fb644d3837e466ef8f/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:289 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-387 DeviceMajor:0 DeviceMinor:387 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92f93306-6430-45d2-887a-f056426c57ed/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:247 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/d35b6637-df31-4561-9b9e-ebf305bd997f/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:345 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cd87de0b-e92c-4740-ab77-b5793a227d40/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:352 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-60 DeviceMajor:0 DeviceMinor:60 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/64fed540-f4cb-4e19-b3d9-7531e486c4f5/volumes/kubernetes.io~projected/kube-api-access-6f5xs DeviceMajor:0 DeviceMinor:55 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:296 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/447f1a32-218e-4599-9a99-011281767a95/volumes/kubernetes.io~projected/kube-api-access-wtg4z DeviceMajor:0 DeviceMinor:388 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f7d22d34d843f1d05143e3db04aa79f40ee8c7db6b98388ee960d3fb761158b3/userdata/shm DeviceMajor:0 DeviceMinor:468 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-232 DeviceMajor:0 DeviceMinor:232 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/185b41a3-2575-4865-ba56-77b1c1b40989/volumes/kubernetes.io~projected/kube-api-access-qwnl8 DeviceMajor:0 DeviceMinor:239 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eb1042dd-c449-4c9e-9a56-899e68938400/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:240 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/788e0688-d9fb-48af-993d-c3bc6a0bc4c5/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:466 Capacity:31985729536 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-488 DeviceMajor:0 DeviceMinor:488 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0babb2305f4b49476ad7ebbb00897318f2c9c7d47ab2b20911f5de33d51d47e2/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:01805ad6a87c74b MacAddress:ca:86:91:60:c0:81 Speed:10000 Mtu:8901} {Name:2a860db12e701c6 MacAddress:1a:4b:04:b4:4c:4c Speed:10000 Mtu:8901} {Name:37eb554ab8065a9 MacAddress:46:a5:f1:3a:05:e9 Speed:10000 Mtu:8901} {Name:3e3baf4bccdfd56 MacAddress:7e:c2:c0:e5:2f:69 Speed:10000 Mtu:8901} {Name:598d4aa00117647 MacAddress:2e:cb:bf:87:67:fb Speed:10000 Mtu:8901} {Name:731a4a90ddf12d0 MacAddress:1e:23:a7:a4:05:c1 Speed:10000 Mtu:8901} {Name:75c8b94a87abd92 MacAddress:2a:ff:d8:a0:07:e3 Speed:10000 Mtu:8901} {Name:7b263ef59b55440 MacAddress:4e:30:8a:68:00:e0 Speed:10000 Mtu:8901} {Name:7dac5eb65ad2969 MacAddress:36:98:20:9f:94:53 Speed:10000 Mtu:8901} {Name:89a4d82ffb50b91 MacAddress:a2:34:c5:af:7c:30 Speed:10000 Mtu:8901} {Name:994dfd7f3fb3997 MacAddress:b2:84:d0:e3:0c:ae Speed:10000 Mtu:8901} {Name:a53af0274c50215 MacAddress:c2:24:23:d3:04:c8 Speed:10000 Mtu:8901} {Name:a94d6d3495d5214 MacAddress:d2:97:18:c9:af:6c Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ff:6a:fd:ed Speed:0 Mtu:9001} {Name:br-int MacAddress:46:43:fc:e0:ce:16 Speed:0 Mtu:8901} {Name:cc03cc009a1e0fb MacAddress:66:d9:e3:e2:15:09 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ff:6a:fd:ed Speed:0 Mtu:9001} {Name:f7d22d34d843f1d MacAddress:1e:55:1c:4b:66:38 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:8e:50:47:74:45:ea Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:66:38:4b:dc:ef:f8 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164328960 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354169 6966 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354210 6966 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354566 6966 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354591 6966 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354718 6966 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354725 6966 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354735 6966 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354753 6966 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.354819 6966 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355002 6966 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355108 6966 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355141 6966 kubelet.go:493] "Attempting to sync node with API server" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355152 6966 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355161 6966 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355170 6966 kubelet.go:405] "Adding apiserver pod source" Oct 30 17:57:17.355390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.355177 6966 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 17:57:17.356386 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.356373 6966 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:57:17.356386 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.356385 6966 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:17.356835 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.356817 6966 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 17:57:17.359473 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359448 6966 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 17:57:17.359529 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359477 6966 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 17:57:17.359594 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359579 6966 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 17:57:17.359776 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359765 6966 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 17:57:17.359806 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359795 6966 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 17:57:17.359891 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359881 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 17:57:17.359923 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359900 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 17:57:17.359923 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359913 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 17:57:17.359976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359927 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 17:57:17.359976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359940 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 17:57:17.359976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359954 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 17:57:17.359976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359967 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359980 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.359994 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.360009 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.360028 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.360038 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.360063 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 17:57:17.360090 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.360073 6966 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 17:57:17.363432 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.363420 6966 server.go:1263] "Started kubelet" Oct 30 17:57:17.363515 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.363482 6966 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 17:57:17.364452 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.363580 6966 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 17:57:17.364452 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.363719 6966 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 17:57:17.364452 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.364051 6966 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 17:57:17.364117 ip-10-0-136-241 systemd[1]: Started Kubernetes Kubelet. Oct 30 17:57:17.365333 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.364964 6966 server.go:317] "Adding debug handlers to kubelet server" Oct 30 17:57:17.369941 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.369904 6966 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 17:57:17.369941 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.369934 6966 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 17:57:17.370100 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.370009 6966 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:51:07 +0000 UTC" deadline="2028-05-01 15:49:16.557031297 +0000 UTC" Oct 30 17:57:17.370100 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.370028 6966 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13173h51m59.187006762s" Oct 30 17:57:17.370100 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.370061 6966 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 17:57:17.370100 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.370073 6966 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 17:57:17.370280 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.370149 6966 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 17:57:17.370888 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.370666 6966 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-241.ec2.internal\" not found" Oct 30 17:57:17.371916 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.371846 6966 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 17:57:17.371916 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.371864 6966 factory.go:55] Registering systemd factory Oct 30 17:57:17.371916 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.371876 6966 factory.go:223] Registration of the systemd container factory successfully Oct 30 17:57:17.372133 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.372114 6966 factory.go:153] Registering CRI-O factory Oct 30 17:57:17.372133 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.372131 6966 factory.go:223] Registration of the crio container factory successfully Oct 30 17:57:17.372295 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.372157 6966 factory.go:103] Registering Raw factory Oct 30 17:57:17.372295 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.372168 6966 manager.go:1199] Started watching for new ooms in manager Oct 30 17:57:17.372520 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.372508 6966 manager.go:319] Starting recovery of all containers Oct 30 17:57:17.373263 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373227 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-grpc-tls" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373263 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373276 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4303592-c23b-49bc-a242-dfefafe43d0a" volumeName="kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-daemon-config" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373288 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b2e7b1-1141-4cb4-a0fa-5376cf8014d8" volumeName="kubernetes.io/empty-dir/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-data-volume" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373298 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373309 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc0cafd7-e383-4391-8d85-073289c3f442" volumeName="kubernetes.io/projected/bc0cafd7-e383-4391-8d85-073289c3f442-kube-api-access-7q4kz" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373320 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4303592-c23b-49bc-a242-dfefafe43d0a" volumeName="kubernetes.io/projected/f4303592-c23b-49bc-a242-dfefafe43d0a-kube-api-access-kxbpv" seLinuxMountContext="" Oct 30 17:57:17.373339 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373331 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="447f1a32-218e-4599-9a99-011281767a95" volumeName="kubernetes.io/projected/447f1a32-218e-4599-9a99-011281767a95-kube-api-access-wtg4z" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373344 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64fed540-f4cb-4e19-b3d9-7531e486c4f5" volumeName="kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373362 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="788e0688-d9fb-48af-993d-c3bc6a0bc4c5" volumeName="kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373374 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f93306-6430-45d2-887a-f056426c57ed" volumeName="kubernetes.io/empty-dir/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-textfile" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373385 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f93306-6430-45d2-887a-f056426c57ed" volumeName="kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-tls" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373397 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f93306-6430-45d2-887a-f056426c57ed" volumeName="kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373408 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373423 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373437 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699b6cb3-446f-440e-8d68-66c4e09ee322" volumeName="kubernetes.io/projected/699b6cb3-446f-440e-8d68-66c4e09ee322-kube-api-access-v8rqz" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373448 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="185b41a3-2575-4865-ba56-77b1c1b40989" volumeName="kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373496 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cf0ad50-df01-43e0-bb53-7c51bbcef6b4" volumeName="kubernetes.io/projected/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kube-api-access-z8tjv" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373509 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d111489-3f0f-49bb-9de2-f400960f01d7" volumeName="kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-env-overrides" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373522 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373535 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373547 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373559 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbd7a723-acf0-4c7c-95a5-2bd54d65460d" volumeName="kubernetes.io/projected/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-kube-api-access-tbst6" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373571 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64fed540-f4cb-4e19-b3d9-7531e486c4f5" volumeName="kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-binary-copy" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373593 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/projected/cd87de0b-e92c-4740-ab77-b5793a227d40-kube-api-access-4r5s8" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373609 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfcad2e0-540b-4e42-ba77-303e060d37bd" volumeName="kubernetes.io/configmap/cfcad2e0-540b-4e42-ba77-303e060d37bd-konnectivity-ca" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373626 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373640 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373658 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1042dd-c449-4c9e-9a56-899e68938400" volumeName="kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-metrics-client-ca" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373670 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373683 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790" volumeName="kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-tmp" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373694 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790" volumeName="kubernetes.io/projected/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-kube-api-access-s9jjx" seLinuxMountContext="" Oct 30 17:57:17.373709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373705 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc0cafd7-e383-4391-8d85-073289c3f442" volumeName="kubernetes.io/secret/bc0cafd7-e383-4391-8d85-073289c3f442-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373724 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/empty-dir/cd87de0b-e92c-4740-ab77-b5793a227d40-audit-log" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373737 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/projected/d35b6637-df31-4561-9b9e-ebf305bd997f-kube-api-access-wfv4g" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373752 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373765 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1042dd-c449-4c9e-9a56-899e68938400" volumeName="kubernetes.io/empty-dir/eb1042dd-c449-4c9e-9a56-899e68938400-volume-directive-shadow" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373776 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="185b41a3-2575-4865-ba56-77b1c1b40989" volumeName="kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373787 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373800 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699b6cb3-446f-440e-8d68-66c4e09ee322" volumeName="kubernetes.io/secret/699b6cb3-446f-440e-8d68-66c4e09ee322-metrics-tls" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373811 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc87edf-5705-49bc-abe7-7cf921ed75ab" volumeName="kubernetes.io/projected/9cc87edf-5705-49bc-abe7-7cf921ed75ab-kube-api-access-h6pm8" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373822 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d111489-3f0f-49bb-9de2-f400960f01d7" volumeName="kubernetes.io/projected/9d111489-3f0f-49bb-9de2-f400960f01d7-kube-api-access-xlpnz" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373837 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1042dd-c449-4c9e-9a56-899e68938400" volumeName="kubernetes.io/projected/eb1042dd-c449-4c9e-9a56-899e68938400-kube-api-access-vp6xh" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373849 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373861 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373873 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373884 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790" volumeName="kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-tuned" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373895 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b91437e-1659-46b1-bc95-687ddd06c392" volumeName="kubernetes.io/configmap/5b91437e-1659-46b1-bc95-687ddd06c392-iptables-alerter-script" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373911 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f93306-6430-45d2-887a-f056426c57ed" volumeName="kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373926 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373937 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373949 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="447f1a32-218e-4599-9a99-011281767a95" volumeName="kubernetes.io/secret/447f1a32-218e-4599-9a99-011281767a95-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373962 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d111489-3f0f-49bb-9de2-f400960f01d7" volumeName="kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-script-lib" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373974 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373986 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.373997 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bbd7a723-acf0-4c7c-95a5-2bd54d65460d" volumeName="kubernetes.io/empty-dir/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-tmp-dir" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374040 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374064 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374078 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374090 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="185b41a3-2575-4865-ba56-77b1c1b40989" volumeName="kubernetes.io/configmap/185b41a3-2575-4865-ba56-77b1c1b40989-metrics-client-ca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374102 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699b6cb3-446f-440e-8d68-66c4e09ee322" volumeName="kubernetes.io/empty-dir/699b6cb3-446f-440e-8d68-66c4e09ee322-tmp-dir" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374114 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b2e7b1-1141-4cb4-a0fa-5376cf8014d8" volumeName="kubernetes.io/projected/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-api-access-m8mhq" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374125 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374142 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374158 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d111489-3f0f-49bb-9de2-f400960f01d7" volumeName="kubernetes.io/secret/9d111489-3f0f-49bb-9de2-f400960f01d7-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374170 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374183 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374195 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/configmap/d35b6637-df31-4561-9b9e-ebf305bd997f-metrics-client-ca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374206 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374218 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374234 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4303592-c23b-49bc-a242-dfefafe43d0a" volumeName="kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-cni-binary-copy" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374250 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374264 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64fed540-f4cb-4e19-b3d9-7531e486c4f5" volumeName="kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374286 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="64fed540-f4cb-4e19-b3d9-7531e486c4f5" volumeName="kubernetes.io/projected/64fed540-f4cb-4e19-b3d9-7531e486c4f5-kube-api-access-6f5xs" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374296 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="699b6cb3-446f-440e-8d68-66c4e09ee322" volumeName="kubernetes.io/configmap/699b6cb3-446f-440e-8d68-66c4e09ee322-config-volume" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374308 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f93306-6430-45d2-887a-f056426c57ed" volumeName="kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-metrics-client-ca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374320 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d111489-3f0f-49bb-9de2-f400960f01d7" volumeName="kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374333 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374346 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374356 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" volumeName="kubernetes.io/projected/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-kube-api-access-l9s6l" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374368 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374387 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374398 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/empty-dir/d35b6637-df31-4561-9b9e-ebf305bd997f-tmp" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374410 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d35b6637-df31-4561-9b9e-ebf305bd997f" volumeName="kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374425 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1042dd-c449-4c9e-9a56-899e68938400" volumeName="kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374439 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374450 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374478 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6990713-1f8a-4ecc-9279-9e28d2c0cba4" volumeName="kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374490 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5b91437e-1659-46b1-bc95-687ddd06c392" volumeName="kubernetes.io/projected/5b91437e-1659-46b1-bc95-687ddd06c392-kube-api-access-qgg2b" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374512 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b2e7b1-1141-4cb4-a0fa-5376cf8014d8" volumeName="kubernetes.io/secret/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374523 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2d95db0-3d14-46b9-99d5-902c2f740585" volumeName="kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374536 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374552 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374569 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="aa838616-f66c-424f-bbd8-eab8cc77ea9c" volumeName="kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374581 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374593 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374604 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4652dca2-9b46-43e6-87d4-70c094ecf09a" volumeName="kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374620 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374633 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cd87de0b-e92c-4740-ab77-b5793a227d40" volumeName="kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-client-ca-bundle" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374648 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cfcad2e0-540b-4e42-ba77-303e060d37bd" volumeName="kubernetes.io/secret/cfcad2e0-540b-4e42-ba77-303e060d37bd-agent-certs" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374665 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e23038e0-b513-49b0-bf52-f66a54d0c3d7" volumeName="kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374679 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1042dd-c449-4c9e-9a56-899e68938400" volumeName="kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374690 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eb1042dd-c449-4c9e-9a56-899e68938400" volumeName="kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374702 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="185b41a3-2575-4865-ba56-77b1c1b40989" volumeName="kubernetes.io/projected/185b41a3-2575-4865-ba56-77b1c1b40989-kube-api-access-qwnl8" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374725 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b" volumeName="kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374737 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85b2e7b1-1141-4cb4-a0fa-5376cf8014d8" volumeName="kubernetes.io/configmap/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374749 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92f93306-6430-45d2-887a-f056426c57ed" volumeName="kubernetes.io/projected/92f93306-6430-45d2-887a-f056426c57ed-kube-api-access-g9szl" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374765 6966 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9cc87edf-5705-49bc-abe7-7cf921ed75ab" volumeName="kubernetes.io/configmap/9cc87edf-5705-49bc-abe7-7cf921ed75ab-serviceca" seLinuxMountContext="" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374777 6966 reconstruct.go:97] "Volume reconstruction finished" Oct 30 17:57:17.374857 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.374784 6966 reconciler.go:29] "Reconciler: start to sync state" Oct 30 17:57:17.403238 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.403154 6966 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 17:57:17.404880 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.404860 6966 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 17:57:17.404880 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.404880 6966 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 17:57:17.405041 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.404900 6966 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 17:57:17.405041 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.404976 6966 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 17:57:17.415774 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.415753 6966 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 17:57:17.416108 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.416083 6966 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:17.416232 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.416164 6966 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:17.425316 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.425288 6966 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="0df97850265b3d6850dc5a026354060e293c3227db38d7c0a3afb9157bcccb3f" exitCode=0 Oct 30 17:57:17.425390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.425322 6966 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="e58660cec6434f04f2c0dbde7cf81ff271bd180aa5097c9f746b37673843b408" exitCode=0 Oct 30 17:57:17.425390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.425333 6966 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="92545f620a73dd66ad7a2fd9077f9cb513eedb172efb5e31a8f4407847b8f3b7" exitCode=0 Oct 30 17:57:17.425390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.425344 6966 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="d741d06c0b470b90a019f20ae0cf6e249cfbc3bd76a1cb3fef2c72eb87a98ca7" exitCode=0 Oct 30 17:57:17.425390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.425352 6966 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="fadf7477261abef9ff48a41d3c6a663a6462bf43769a00c7fa227e28c5a8e72c" exitCode=0 Oct 30 17:57:17.425390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.425361 6966 generic.go:356] "Generic (PLEG): container finished" podID="64fed540-f4cb-4e19-b3d9-7531e486c4f5" containerID="d844be6efa2643ea424fe8e728dc969cdbecb922abd4e80fbaf82ebbc9f068af" exitCode=0 Oct 30 17:57:17.427301 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.427274 6966 generic.go:356] "Generic (PLEG): container finished" podID="181ed97c8a1c8f7ef8e42a31d315dd0f" containerID="68d636785d8954011199b11568c848daf74900e9051dac0eefefa0827f70699b" exitCode=0 Oct 30 17:57:17.438713 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.438692 6966 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:17.438801 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.438743 6966 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 17:57:17.461553 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.461532 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" exitCode=0 Oct 30 17:57:17.468061 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.468035 6966 generic.go:356] "Generic (PLEG): container finished" podID="92f93306-6430-45d2-887a-f056426c57ed" containerID="dfc9ae5a990c8217fcf69369b3115e54d6029bb134d016afe4f5ec4d69236026" exitCode=0 Oct 30 17:57:17.491783 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.491721 6966 manager.go:324] Recovery completed Oct 30 17:57:17.505246 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.505223 6966 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 17:57:17.514286 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514267 6966 cpu_manager.go:226] "Starting" policy="none" Oct 30 17:57:17.514286 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514282 6966 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 17:57:17.514423 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514301 6966 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 17:57:17.514493 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514480 6966 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 17:57:17.514557 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514498 6966 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 17:57:17.514557 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514523 6966 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 17:57:17.514557 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514531 6966 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 17:57:17.514557 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514539 6966 policy_none.go:50] "Start" Oct 30 17:57:17.514557 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514552 6966 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 17:57:17.514776 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.514566 6966 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 17:57:17.515835 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.515820 6966 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 17:57:17.515903 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.515838 6966 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 17:57:17.515903 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.515849 6966 policy_none.go:44] "Start" Oct 30 17:57:17.520486 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.520446 6966 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 17:57:17.524912 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.524900 6966 manager.go:341] "Starting Device Plugin manager" Oct 30 17:57:17.525569 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.524924 6966 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 17:57:17.525651 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.525592 6966 server.go:87] "Starting device plugin registration server" Oct 30 17:57:17.525958 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.525937 6966 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 17:57:17.526032 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.525960 6966 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 17:57:17.526199 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.526180 6966 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 17:57:17.526337 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.526283 6966 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 17:57:17.526337 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.526334 6966 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 17:57:17.526337 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.526342 6966 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 17:57:17.529390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.526491 6966 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T17:57:17.526314728Z","UUID":"d95bd5eb-47d3-4725-bdc3-5edb80af235f","Handler":null,"Name":"","Endpoint":""} Oct 30 17:57:17.529390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.528163 6966 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 17:57:17.529390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.528185 6966 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 17:57:17.529390 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.529060 6966 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 17:57:17.626175 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.626159 6966 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 17:57:17.627904 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.627881 6966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientMemory" Oct 30 17:57:17.627970 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.627914 6966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 17:57:17.627970 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.627925 6966 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-241.ec2.internal" event="NodeHasSufficientPID" Oct 30 17:57:17.627970 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.627966 6966 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.660384 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.660363 6966 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.660441 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.660411 6966 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.705662 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.705628 6966 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal"] Oct 30 17:57:17.706236 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706223 6966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.706288 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706238 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" event={"ID":"181ed97c8a1c8f7ef8e42a31d315dd0f","Type":"ContainerStarted","Data":"a98d29d884af12a100a367a8cfa55856384f7fbb821a63efe82a45a50749a14b"} Oct 30 17:57:17.706288 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706279 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" event={"ID":"181ed97c8a1c8f7ef8e42a31d315dd0f","Type":"ContainerDied","Data":"68d636785d8954011199b11568c848daf74900e9051dac0eefefa0827f70699b"} Oct 30 17:57:17.706352 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706300 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" event={"ID":"181ed97c8a1c8f7ef8e42a31d315dd0f","Type":"ContainerStarted","Data":"51084b9baa34e0f81e998be63eee63383790f6dd86c3b823cf32306f30036312"} Oct 30 17:57:17.706352 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706346 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" event={"ID":"ef2a11081781848cec6e4c7eceee10f4","Type":"ContainerStarted","Data":"aab41c539eec2087ae934b6ad28636b4d398c9c9ba8c316b1d20055b88e2d6b4"} Oct 30 17:57:17.706422 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706364 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" event={"ID":"ef2a11081781848cec6e4c7eceee10f4","Type":"ContainerStarted","Data":"9298f9079bcb0dd9196f554cc583a63268264d6913998612ad79fcbe73d05851"} Oct 30 17:57:17.706491 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.706456 6966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.729145 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.729127 6966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:17.729206 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.729164 6966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.729809 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.729796 6966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:17.729855 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:17.729830 6966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.818141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.818122 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2a11081781848cec6e4c7eceee10f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" (UID: \"ef2a11081781848cec6e4c7eceee10f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.818206 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.818146 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.818206 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.818175 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.918691 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.918668 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.918755 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.918695 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.918755 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.918745 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2a11081781848cec6e4c7eceee10f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" (UID: \"ef2a11081781848cec6e4c7eceee10f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.918860 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.918758 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ef2a11081781848cec6e4c7eceee10f4-config\") pod \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" (UID: \"ef2a11081781848cec6e4c7eceee10f4\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.918860 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.918767 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:17.918860 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:17.918819 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/181ed97c8a1c8f7ef8e42a31d315dd0f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" (UID: \"181ed97c8a1c8f7ef8e42a31d315dd0f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:18.355688 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.355663 6966 apiserver.go:52] "Watching apiserver" Oct 30 17:57:18.405235 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.405211 6966 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 17:57:18.405935 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.405912 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-56qxs","openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv","openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq","openshift-multus/network-metrics-daemon-c8fkb","openshift-multus/multus-tg6rw","openshift-network-diagnostics/network-check-target-pjfgc","kube-system/global-pull-secret-syncer-gqtbs","openshift-console/console-f9d75c56b-xtlgz","openshift-dns/dns-default-z5c26","openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p","openshift-ovn-kubernetes/ovnkube-node-brcs6","openshift-console/console-fb8fd8fd8-5ks86","openshift-ingress-canary/ingress-canary-vm92t","openshift-monitoring/node-exporter-4fgr8","openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf","openshift-monitoring/thanos-querier-67bc75cc55-5scfq","kube-system/konnectivity-agent-bzkm6","openshift-cluster-node-tuning-operator/tuned-jhj27","openshift-image-registry/node-ca-rd5c5","openshift-console/console-6ccccdd4d8-bz6n6","openshift-dns/node-resolver-l2ddq","openshift-multus/multus-additional-cni-plugins-cpntx","openshift-network-operator/iptables-alerter-7llnl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58","openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:18.422385 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422364 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.422656 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422605 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 17:57:18.422656 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422631 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 17:57:18.422814 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422664 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.422814 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422675 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 17:57:18.422814 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422735 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422830 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422839 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422914 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422944 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422947 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422918 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-bkbt2\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422955 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422921 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-tzlmk\"" Oct 30 17:57:18.422976 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422955 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.422997 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423010 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423010 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423038 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423039 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-z6vc6\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423106 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-8vdz2\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423177 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-26whl\"" Oct 30 17:57:18.423296 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423242 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z9xqn\"" Oct 30 17:57:18.423584 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.423312 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-v7j5r\"" Oct 30 17:57:18.431612 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.431586 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 17:57:18.432176 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.432056 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.432176 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.432101 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.432176 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.432148 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.432176 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.432170 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pc6fd\"" Oct 30 17:57:18.432640 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.432625 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 17:57:18.432774 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.432760 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.433081 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.433049 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7j6rv\"" Oct 30 17:57:18.433081 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.433065 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.433081 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.433083 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.433228 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.433116 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 17:57:18.433228 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.433130 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 17:57:18.433326 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.433311 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 17:57:18.434217 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.434204 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.435392 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.435377 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.435573 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.435560 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.435904 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.435891 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 17:57:18.436412 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436391 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 17:57:18.436646 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436629 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.436646 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436642 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.436722 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436689 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 17:57:18.436761 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436748 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.436806 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436788 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.436963 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.436948 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 17:57:18.437147 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437037 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:57:18.437147 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437084 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.437147 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437124 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 17:57:18.437300 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437165 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2dl6z\"" Oct 30 17:57:18.437300 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437166 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:57:18.437300 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437233 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:57:18.437300 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437248 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 17:57:18.437454 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437301 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:57:18.437698 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437681 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vkdz6\"" Oct 30 17:57:18.437748 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437696 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dxtgs\"" Oct 30 17:57:18.437792 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437776 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 17:57:18.437977 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.437959 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-ksmnn\"" Oct 30 17:57:18.438035 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438018 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:57:18.438035 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438027 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438039 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438060 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438073 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438093 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438113 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438116 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438120 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 17:57:18.438141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438137 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-c2g6c\"" Oct 30 17:57:18.438567 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438547 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:57:18.438819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438799 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:57:18.438819 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438805 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dqs6z\"" Oct 30 17:57:18.438956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438824 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-zxlb9\"" Oct 30 17:57:18.438956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438848 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 17:57:18.438956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438863 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 17:57:18.438956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438852 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 17:57:18.438956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438879 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 17:57:18.438956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.438885 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 17:57:18.443691 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443669 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-dhnvk\"" Oct 30 17:57:18.443691 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443672 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 17:57:18.443800 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443782 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t7pkc\"" Oct 30 17:57:18.443847 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443789 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:57:18.444006 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443988 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-d6gl7\"" Oct 30 17:57:18.444073 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443996 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 17:57:18.444073 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.443996 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-fc3pb24n385dj\"" Oct 30 17:57:18.444073 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444035 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 17:57:18.444073 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444040 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-9zbdg\"" Oct 30 17:57:18.444073 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444039 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 17:57:18.444073 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444058 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 17:57:18.444310 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444161 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 17:57:18.444310 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444208 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 17:57:18.444801 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444789 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 17:57:18.444839 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.444815 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 17:57:18.448183 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448171 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cclgx\"" Oct 30 17:57:18.448252 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448220 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-5zvv9\"" Oct 30 17:57:18.448305 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448269 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 17:57:18.448305 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448277 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:57:18.448305 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448293 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mmrm7\"" Oct 30 17:57:18.448429 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448417 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 17:57:18.448494 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448417 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-fbpgnt1o3ru57\"" Oct 30 17:57:18.448538 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.448503 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 17:57:18.449311 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.449295 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 17:57:18.473804 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.473783 6966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:18.474282 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.474263 6966 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:18.475891 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.475873 6966 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 17:57:18.496403 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.496386 6966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:18.496454 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:18.496424 6966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-241.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-241.ec2.internal" Oct 30 17:57:18.496980 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.496968 6966 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 17:57:18.497017 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:18.496997 6966 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-241.ec2.internal" Oct 30 17:57:18.520908 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.520883 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-metrics-server-audit-profiles\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.520998 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.520917 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfcad2e0-540b-4e42-ba77-303e060d37bd-konnectivity-ca\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:57:18.520998 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.520935 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-sys\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.520998 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.520952 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.520998 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.520974 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rqz\" (UniqueName: \"kubernetes.io/projected/699b6cb3-446f-440e-8d68-66c4e09ee322-kube-api-access-v8rqz\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.521198 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521071 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.521198 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521105 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-etc-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.521198 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521135 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-multus\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.521198 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521163 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-kubelet\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.521198 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521194 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cfcad2e0-540b-4e42-ba77-303e060d37bd-konnectivity-ca\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521209 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521213 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-metrics-server-audit-profiles\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521234 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxbpv\" (UniqueName: \"kubernetes.io/projected/f4303592-c23b-49bc-a242-dfefafe43d0a-kube-api-access-kxbpv\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521255 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521331 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521354 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.521382 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521373 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfv4g\" (UniqueName: \"kubernetes.io/projected/d35b6637-df31-4561-9b9e-ebf305bd997f-kube-api-access-wfv4g\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521397 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521421 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521447 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-config\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521497 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-system-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521519 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-netns\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521541 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-tmp\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521564 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521564 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521619 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521651 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-tmp\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.521674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521661 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cd87de0b-e92c-4740-ab77-b5793a227d40-audit-log\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521698 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-device-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521711 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-config\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521748 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/cd87de0b-e92c-4740-ab77-b5793a227d40-audit-log\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521758 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521778 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521788 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cnibin\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521825 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521828 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-original-pull-secret\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521825 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521834 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-log-socket\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521872 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9s6l\" (UniqueName: \"kubernetes.io/projected/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-kube-api-access-l9s6l\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521929 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-etc-kubernetes\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521970 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.521972 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-root\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522007 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9szl\" (UniqueName: \"kubernetes.io/projected/92f93306-6430-45d2-887a-f056426c57ed-kube-api-access-g9szl\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522034 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522062 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-script-lib\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522088 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.522151 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522112 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-metrics-client-ca\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522220 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-systemd\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522256 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-var-lib-kubelet\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522261 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522296 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522312 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522312 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-ovnkube-script-lib\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522334 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522340 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-metrics-client-ca\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522368 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522394 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-conf\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522418 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522442 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-wtmp\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522492 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-hosts-file\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522516 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522578 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522584 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522617 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6f5xs\" (UniqueName: \"kubernetes.io/projected/64fed540-f4cb-4e19-b3d9-7531e486c4f5-kube-api-access-6f5xs\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522642 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb1042dd-c449-4c9e-9a56-899e68938400-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522668 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-kubelet-config\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522692 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-kubelet\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522702 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522716 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522738 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/eb1042dd-c449-4c9e-9a56-899e68938400-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.522902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522824 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522950 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522970 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.522983 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-client-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523086 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523103 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-node-log\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523116 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523124 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523159 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523186 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523212 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523220 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-client-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523237 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d35b6637-df31-4561-9b9e-ebf305bd997f-tmp\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523261 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523264 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523349 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp6xh\" (UniqueName: \"kubernetes.io/projected/eb1042dd-c449-4c9e-9a56-899e68938400-kube-api-access-vp6xh\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523377 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-tls\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523353 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d35b6637-df31-4561-9b9e-ebf305bd997f-tmp\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523402 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-kubernetes\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523427 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-systemd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523429 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523441 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-metrics-certs\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523501 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523527 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-cnibin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523562 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523599 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523615 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523618 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-tls\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523704 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-client-certs\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523729 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523745 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-host\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523754 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523761 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-netns\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523786 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-data-volume\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523820 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523848 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523884 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-data-volume\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.523885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523889 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523919 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/cd87de0b-e92c-4740-ab77-b5793a227d40-secret-metrics-server-client-certs\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523928 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd87de0b-e92c-4740-ab77-b5793a227d40-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523955 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q4kz\" (UniqueName: \"kubernetes.io/projected/bc0cafd7-e383-4391-8d85-073289c3f442-kube-api-access-7q4kz\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.523973 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/447f1a32-218e-4599-9a99-011281767a95-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524005 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-accelerators-collector-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524012 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524022 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524036 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524053 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-daemon-config\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524072 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524088 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524110 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524130 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524147 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524149 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/447f1a32-218e-4599-9a99-011281767a95-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524163 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pm8\" (UniqueName: \"kubernetes.io/projected/9cc87edf-5705-49bc-abe7-7cf921ed75ab-kube-api-access-h6pm8\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524183 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8tjv\" (UniqueName: \"kubernetes.io/projected/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kube-api-access-z8tjv\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524195 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524202 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-socket-dir-parent\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524222 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524231 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbst6\" (UniqueName: \"kubernetes.io/projected/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-kube-api-access-tbst6\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524267 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524294 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91437e-1659-46b1-bc95-687ddd06c392-iptables-alerter-script\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524299 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524379 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524475 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524485 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5b91437e-1659-46b1-bc95-687ddd06c392-iptables-alerter-script\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524496 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524522 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-daemon-config\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524526 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91437e-1659-46b1-bc95-687ddd06c392-host-slash\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524617 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524660 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524655 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524754 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-conf-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524791 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524803 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524816 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524843 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-hostroot\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524868 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-multus-certs\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524912 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185b41a3-2575-4865-ba56-77b1c1b40989-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524940 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-sys\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524954 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.524986 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525032 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jjx\" (UniqueName: \"kubernetes.io/projected/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-kube-api-access-s9jjx\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525118 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d111489-3f0f-49bb-9de2-f400960f01d7-ovn-node-metrics-cert\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525151 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525160 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/185b41a3-2575-4865-ba56-77b1c1b40989-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525197 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525230 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525165 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525251 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-tuned\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525277 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525303 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525327 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525375 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525391 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/9d111489-3f0f-49bb-9de2-f400960f01d7-ovn-node-metrics-cert\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525421 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525445 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525393 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-tuned\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525449 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cc87edf-5705-49bc-abe7-7cf921ed75ab-host\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525509 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cc87edf-5705-49bc-abe7-7cf921ed75ab-serviceca\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525525 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525528 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.525527 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525532 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwnl8\" (UniqueName: \"kubernetes.io/projected/185b41a3-2575-4865-ba56-77b1c1b40989-kube-api-access-qwnl8\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525591 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525618 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699b6cb3-446f-440e-8d68-66c4e09ee322-tmp-dir\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525627 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-binary-copy\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525635 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525640 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525644 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtg4z\" (UniqueName: \"kubernetes.io/projected/447f1a32-218e-4599-9a99-011281767a95-kube-api-access-wtg4z\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525694 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-run\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525718 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-env-overrides\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525696 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/699b6cb3-446f-440e-8d68-66c4e09ee322-tmp-dir\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525768 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8mhq\" (UniqueName: \"kubernetes.io/projected/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-api-access-m8mhq\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525902 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-k8s-cni-cncf-io\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525909 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/9d111489-3f0f-49bb-9de2-f400960f01d7-env-overrides\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.525982 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-textfile\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526006 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526023 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-slash\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526040 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-crio-socket\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526024 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-textfile\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526080 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526114 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526114 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9cc87edf-5705-49bc-abe7-7cf921ed75ab-serviceca\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526155 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526164 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526196 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-modprobe-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526221 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysconfig\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526249 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526273 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-dbus\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526267 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/185b41a3-2575-4865-ba56-77b1c1b40989-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526318 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-systemd-units\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526351 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-var-lib-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526382 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-bin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526408 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/699b6cb3-446f-440e-8d68-66c4e09ee322-metrics-tls\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526451 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-grpc-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526454 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526503 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfcad2e0-540b-4e42-ba77-303e060d37bd-agent-certs\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526520 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-lib-modules\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526537 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-netd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526571 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/699b6cb3-446f-440e-8d68-66c4e09ee322-metrics-tls\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526613 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlpnz\" (UniqueName: \"kubernetes.io/projected/9d111489-3f0f-49bb-9de2-f400960f01d7-kube-api-access-xlpnz\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526641 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526692 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-grpc-tls\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526734 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526762 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-tmp-dir\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526785 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cfcad2e0-540b-4e42-ba77-303e060d37bd-agent-certs\") pod \"konnectivity-agent-bzkm6\" (UID: \"cfcad2e0-540b-4e42-ba77-303e060d37bd\") " pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526786 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526818 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526839 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526899 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-tmp-dir\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526920 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526932 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526943 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526974 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5s8\" (UniqueName: \"kubernetes.io/projected/cd87de0b-e92c-4740-ab77-b5793a227d40-kube-api-access-4r5s8\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.526999 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-tls\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527010 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527105 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d35b6637-df31-4561-9b9e-ebf305bd997f-metrics-client-ca\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527128 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-cni-binary-copy\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527135 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527144 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0cafd7-e383-4391-8d85-073289c3f442-cert\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527144 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527160 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527200 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-ovn\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527209 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-tls\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527227 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-bin\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527276 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/699b6cb3-446f-440e-8d68-66c4e09ee322-config-volume\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527277 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527299 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d35b6637-df31-4561-9b9e-ebf305bd997f-metrics-client-ca\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527317 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f4303592-c23b-49bc-a242-dfefafe43d0a-cni-binary-copy\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527318 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgg2b\" (UniqueName: \"kubernetes.io/projected/5b91437e-1659-46b1-bc95-687ddd06c392-kube-api-access-qgg2b\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527343 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc0cafd7-e383-4391-8d85-073289c3f442-cert\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527408 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527427 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527487 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/eb1042dd-c449-4c9e-9a56-899e68938400-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527512 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/699b6cb3-446f-440e-8d68-66c4e09ee322-config-volume\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527523 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527568 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527668 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527697 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.527891 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d35b6637-df31-4561-9b9e-ebf305bd997f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.528167 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.528001 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.541968 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.541934 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxbpv\" (UniqueName: \"kubernetes.io/projected/f4303592-c23b-49bc-a242-dfefafe43d0a-kube-api-access-kxbpv\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.542095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.542078 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rqz\" (UniqueName: \"kubernetes.io/projected/699b6cb3-446f-440e-8d68-66c4e09ee322-kube-api-access-v8rqz\") pod \"dns-default-z5c26\" (UID: \"699b6cb3-446f-440e-8d68-66c4e09ee322\") " pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:18.542095 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.542089 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9s6l\" (UniqueName: \"kubernetes.io/projected/3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b-kube-api-access-l9s6l\") pod \"network-metrics-daemon-c8fkb\" (UID: \"3f41cfd0-1e7a-4972-9556-0a6b4ed7fc5b\") " pod="openshift-multus/network-metrics-daemon-c8fkb" Oct 30 17:57:18.542300 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.542282 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfv4g\" (UniqueName: \"kubernetes.io/projected/d35b6637-df31-4561-9b9e-ebf305bd997f-kube-api-access-wfv4g\") pod \"thanos-querier-67bc75cc55-5scfq\" (UID: \"d35b6637-df31-4561-9b9e-ebf305bd997f\") " pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:18.542428 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.542413 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6f5xs\" (UniqueName: \"kubernetes.io/projected/64fed540-f4cb-4e19-b3d9-7531e486c4f5-kube-api-access-6f5xs\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.542428 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.542422 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9szl\" (UniqueName: \"kubernetes.io/projected/92f93306-6430-45d2-887a-f056426c57ed-kube-api-access-g9szl\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.544483 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.544450 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q4kz\" (UniqueName: \"kubernetes.io/projected/bc0cafd7-e383-4391-8d85-073289c3f442-kube-api-access-7q4kz\") pod \"ingress-canary-vm92t\" (UID: \"bc0cafd7-e383-4391-8d85-073289c3f442\") " pod="openshift-ingress-canary/ingress-canary-vm92t" Oct 30 17:57:18.545928 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.545911 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp6xh\" (UniqueName: \"kubernetes.io/projected/eb1042dd-c449-4c9e-9a56-899e68938400-kube-api-access-vp6xh\") pod \"kube-state-metrics-5dc8d7d89b-dm4fv\" (UID: \"eb1042dd-c449-4c9e-9a56-899e68938400\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-dm4fv" Oct 30 17:57:18.548030 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.548012 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwnl8\" (UniqueName: \"kubernetes.io/projected/185b41a3-2575-4865-ba56-77b1c1b40989-kube-api-access-qwnl8\") pod \"openshift-state-metrics-7bd7489957-fqklq\" (UID: \"185b41a3-2575-4865-ba56-77b1c1b40989\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-fqklq" Oct 30 17:57:18.548398 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.548371 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbst6\" (UniqueName: \"kubernetes.io/projected/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-kube-api-access-tbst6\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.548712 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pm8\" (UniqueName: \"kubernetes.io/projected/9cc87edf-5705-49bc-abe7-7cf921ed75ab-kube-api-access-h6pm8\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.548728 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8mhq\" (UniqueName: \"kubernetes.io/projected/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-kube-api-access-m8mhq\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.548920 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtg4z\" (UniqueName: \"kubernetes.io/projected/447f1a32-218e-4599-9a99-011281767a95-kube-api-access-wtg4z\") pod \"monitoring-plugin-77f4f76594-bmk6p\" (UID: \"447f1a32-218e-4599-9a99-011281767a95\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549074 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp5mb\" (UniqueName: \"kubernetes.io/projected/a6990713-1f8a-4ecc-9279-9e28d2c0cba4-kube-api-access-gp5mb\") pod \"network-check-target-pjfgc\" (UID: \"a6990713-1f8a-4ecc-9279-9e28d2c0cba4\") " pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549167 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jjx\" (UniqueName: \"kubernetes.io/projected/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-kube-api-access-s9jjx\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549205 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") pod \"console-f9d75c56b-xtlgz\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:18.549255 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549228 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgg2b\" (UniqueName: \"kubernetes.io/projected/5b91437e-1659-46b1-bc95-687ddd06c392-kube-api-access-qgg2b\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.549502 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549364 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5s8\" (UniqueName: \"kubernetes.io/projected/cd87de0b-e92c-4740-ab77-b5793a227d40-kube-api-access-4r5s8\") pod \"metrics-server-5c6d98fbbf-ckgjf\" (UID: \"cd87de0b-e92c-4740-ab77-b5793a227d40\") " pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:18.549502 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549425 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8tjv\" (UniqueName: \"kubernetes.io/projected/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kube-api-access-z8tjv\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.549502 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549431 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") pod \"console-6ccccdd4d8-bz6n6\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:18.549502 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.549426 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") pod \"alertmanager-main-0\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:18.550288 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.550272 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") pod \"console-fb8fd8fd8-5ks86\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:18.554204 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.554187 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlpnz\" (UniqueName: \"kubernetes.io/projected/9d111489-3f0f-49bb-9de2-f400960f01d7-kube-api-access-xlpnz\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628236 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628185 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-etc-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628236 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628212 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-multus\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628236 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628226 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-kubelet\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628242 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628279 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-system-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628294 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-multus\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628299 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-etc-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628317 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-netns\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628340 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-kubelet\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628344 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-system-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628345 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-device-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628371 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-netns\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628382 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-device-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.628399 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628390 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cnibin\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628410 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-log-socket\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628426 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-etc-kubernetes\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628344 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-tuning-conf-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628440 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-root\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628476 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-systemd\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628495 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-var-lib-kubelet\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628500 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-etc-kubernetes\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628488 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-cnibin\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628506 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-log-socket\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628517 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-root\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628514 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628542 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-systemd\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628556 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-conf\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628558 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628564 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-var-lib-kubelet\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628571 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628591 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628615 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-wtmp\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628643 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-hosts-file\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628673 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-conf\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628678 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-kubelet-config\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628696 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-node-exporter-wtmp\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628706 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-kubelet\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628721 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-kubelet-config\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628727 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-kubelet\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628724 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-node-log\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628706 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbd7a723-acf0-4c7c-95a5-2bd54d65460d-hosts-file\") pod \"node-resolver-l2ddq\" (UID: \"bbd7a723-acf0-4c7c-95a5-2bd54d65460d\") " pod="openshift-dns/node-resolver-l2ddq" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628746 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-node-log\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628757 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-kubernetes\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628773 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-systemd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628788 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628781 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-kubernetes\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628806 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-cnibin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628816 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-systemd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628822 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-host\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628854 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-host\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628856 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-netns\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628876 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-run-netns\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628857 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628884 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.628883 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628906 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-socket-dir-parent\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628858 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-cnibin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628922 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91437e-1659-46b1-bc95-687ddd06c392-host-slash\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628938 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628951 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5b91437e-1659-46b1-bc95-687ddd06c392-host-slash\") pod \"iptables-alerter-7llnl\" (UID: \"5b91437e-1659-46b1-bc95-687ddd06c392\") " pod="openshift-network-operator/iptables-alerter-7llnl" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628912 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-sys-fs\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628963 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-socket-dir-parent\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628977 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-conf-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628953 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-conf-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628982 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.628998 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-hostroot\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629014 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-multus-certs\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629027 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-sys\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629043 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629060 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cc87edf-5705-49bc-abe7-7cf921ed75ab-host\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629076 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-run\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629087 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-multus-certs\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629093 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-k8s-cni-cncf-io\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629088 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/92f93306-6430-45d2-887a-f056426c57ed-sys\") pod \"node-exporter-4fgr8\" (UID: \"92f93306-6430-45d2-887a-f056426c57ed\") " pod="openshift-monitoring/node-exporter-4fgr8" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629106 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-hostroot\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629118 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-slash\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629124 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-socket-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629142 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-crio-socket\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629122 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9cc87edf-5705-49bc-abe7-7cf921ed75ab-host\") pod \"node-ca-rd5c5\" (UID: \"9cc87edf-5705-49bc-abe7-7cf921ed75ab\") " pod="openshift-image-registry/node-ca-rd5c5" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629145 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-run\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629152 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-run-k8s-cni-cncf-io\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629164 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629146 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-slash\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629187 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85b2e7b1-1141-4cb4-a0fa-5376cf8014d8-crio-socket\") pod \"insights-runtime-extractor-56qxs\" (UID: \"85b2e7b1-1141-4cb4-a0fa-5376cf8014d8\") " pod="openshift-insights/insights-runtime-extractor-56qxs" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629191 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-modprobe-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629223 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysconfig\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629200 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-etc-selinux\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629242 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629269 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysconfig\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629282 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-dbus\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629286 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-modprobe-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629311 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-systemd-units\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629330 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-etc-sysctl-d\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629316 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/788e0688-d9fb-48af-993d-c3bc6a0bc4c5-dbus\") pod \"global-pull-secret-syncer-gqtbs\" (UID: \"788e0688-d9fb-48af-993d-c3bc6a0bc4c5\") " pod="kube-system/global-pull-secret-syncer-gqtbs" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629338 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-var-lib-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629334 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-systemd-units\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629364 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-var-lib-openvswitch\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629369 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-bin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629398 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-host-var-lib-cni-bin\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629425 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-lib-modules\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629508 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-lib-modules\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629507 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-netd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629539 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629541 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-netd\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629566 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/64fed540-f4cb-4e19-b3d9-7531e486c4f5-system-cni-dir\") pod \"multus-additional-cni-plugins-cpntx\" (UID: \"64fed540-f4cb-4e19-b3d9-7531e486c4f5\") " pod="openshift-multus/multus-additional-cni-plugins-cpntx" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629570 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-ovn\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629591 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-run-ovn\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629608 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-bin\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629634 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629659 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629681 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9d111489-3f0f-49bb-9de2-f400960f01d7-host-cni-bin\") pod \"ovnkube-node-brcs6\" (UID: \"9d111489-3f0f-49bb-9de2-f400960f01d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629700 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-sys\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629709 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9cf0ad50-df01-43e0-bb53-7c51bbcef6b4-registration-dir\") pod \"aws-ebs-csi-driver-node-rqc58\" (UID: \"9cf0ad50-df01-43e0-bb53-7c51bbcef6b4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-rqc58" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629720 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f4303592-c23b-49bc-a242-dfefafe43d0a-multus-cni-dir\") pod \"multus-tg6rw\" (UID: \"f4303592-c23b-49bc-a242-dfefafe43d0a\") " pod="openshift-multus/multus-tg6rw" Oct 30 17:57:18.630254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.629767 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790-sys\") pod \"tuned-jhj27\" (UID: \"49d1faa3-7a92-46c7-a3b1-3b1f4e1f8790\") " pod="openshift-cluster-node-tuning-operator/tuned-jhj27" Oct 30 17:57:18.940060 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:18.939995 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:57:19.034986 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:19.034951 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb8fd8fd8-5ks86"] Oct 30 17:57:19.158502 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:19.158477 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-z5c26" Oct 30 17:57:19.668834 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:19.668806 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:20.009710 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:20.009634 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-67bc75cc55-5scfq" Oct 30 17:57:20.713421 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:20.713382 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:20.806772 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:20.806738 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:21.061772 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:21.061745 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:21.853923 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:21.853894 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:21.854390 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:21.854148 6966 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 17:57:22.725181 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:22.725149 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5c6d98fbbf-ckgjf" Oct 30 17:57:24.033375 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:24.033334 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:24.720243 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:24.720216 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-bmk6p" Oct 30 17:57:25.043194 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:25.043122 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:28.113845 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:28.113817 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-brcs6" Oct 30 17:57:29.003628 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.003555 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-58cbf4966f-qjsxr"] Oct 30 17:57:29.049859 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.049830 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cbf4966f-qjsxr"] Oct 30 17:57:29.049987 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.049915 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.100827 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.100795 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:29.101217 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.101180 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="alertmanager" containerID="cri-o://0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" gracePeriod=120 Oct 30 17:57:29.101376 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.101346 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="prom-label-proxy" containerID="cri-o://1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" gracePeriod=120 Oct 30 17:57:29.101540 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.101195 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-metric" containerID="cri-o://35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" gracePeriod=120 Oct 30 17:57:29.101540 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.101387 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="config-reloader" containerID="cri-o://25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" gracePeriod=120 Oct 30 17:57:29.101540 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.101426 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy" containerID="cri-o://f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" gracePeriod=120 Oct 30 17:57:29.101540 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.101500 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-web" containerID="cri-o://7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" gracePeriod=120 Oct 30 17:57:29.197767 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197742 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c7525d-92a7-400b-8367-b88234b4a79c-tmp\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197775 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-trusted-ca-bundle\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197795 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-service-ca\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197919 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jndv2\" (UniqueName: \"kubernetes.io/projected/b8c7525d-92a7-400b-8367-b88234b4a79c-kube-api-access-jndv2\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197938 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c7525d-92a7-400b-8367-b88234b4a79c-console-serving-cert\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197957 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c7525d-92a7-400b-8367-b88234b4a79c-console-oauth-config\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.197984 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-oauth-serving-cert\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.198071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.198054 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-console-config\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.247385 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.247359 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:29.298902 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.298873 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jndv2\" (UniqueName: \"kubernetes.io/projected/b8c7525d-92a7-400b-8367-b88234b4a79c-kube-api-access-jndv2\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.298993 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.298922 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c7525d-92a7-400b-8367-b88234b4a79c-console-serving-cert\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.298993 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.298957 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c7525d-92a7-400b-8367-b88234b4a79c-console-oauth-config\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.299075 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.298994 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-oauth-serving-cert\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.299075 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.299027 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-console-config\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.299141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.299092 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c7525d-92a7-400b-8367-b88234b4a79c-tmp\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.299141 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.299116 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-trusted-ca-bundle\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.299211 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.299147 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-service-ca\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.300349 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.300008 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-service-ca\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.300349 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.300162 6966 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 17:57:29.300349 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.300346 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-oauth-serving-cert\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.300622 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.300595 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-console-config\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.302157 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.301671 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8c7525d-92a7-400b-8367-b88234b4a79c-trusted-ca-bundle\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.302157 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.301940 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8c7525d-92a7-400b-8367-b88234b4a79c-tmp\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.302728 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.302708 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8c7525d-92a7-400b-8367-b88234b4a79c-console-oauth-config\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.303520 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.303499 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8c7525d-92a7-400b-8367-b88234b4a79c-console-serving-cert\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.322259 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.322239 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jndv2\" (UniqueName: \"kubernetes.io/projected/b8c7525d-92a7-400b-8367-b88234b4a79c-kube-api-access-jndv2\") pod \"console-58cbf4966f-qjsxr\" (UID: \"b8c7525d-92a7-400b-8367-b88234b4a79c\") " pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.373880 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.373860 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:29.393709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.393685 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.459014 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.458146 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:29.501284 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501250 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501284 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501287 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501448 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501316 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501448 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501341 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501448 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501393 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501448 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501432 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501692 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501513 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501692 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501555 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501692 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501584 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501692 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501628 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501692 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501662 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:29.501692 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501682 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501972 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501733 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.501972 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.501768 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") pod \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\" (UID: \"aa838616-f66c-424f-bbd8-eab8cc77ea9c\") " Oct 30 17:57:29.502069 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.502042 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-main-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.502248 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.502222 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:29.502815 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.502722 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:29.504577 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.504538 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.505030 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.504972 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:29.505268 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.505243 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.505804 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.505777 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.505885 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.505831 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:29.505991 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.505968 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.506137 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.506116 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.507172 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.507148 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "kube-api-access-56kz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:29.509770 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.509746 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.516063 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516038 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config" pod "aa838616-f66c-424f-bbd8-eab8cc77ea9c" (UID: "aa838616-f66c-424f-bbd8-eab8cc77ea9c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:29.516590 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516567 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" exitCode=0 Oct 30 17:57:29.516590 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516587 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" exitCode=0 Oct 30 17:57:29.516695 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516595 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" exitCode=0 Oct 30 17:57:29.516695 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516602 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" exitCode=0 Oct 30 17:57:29.516695 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516610 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" exitCode=0 Oct 30 17:57:29.516695 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516615 6966 generic.go:356] "Generic (PLEG): container finished" podID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" exitCode=0 Oct 30 17:57:29.516695 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516647 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.516695 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516649 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516734 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516784 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516807 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516825 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516868 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516910 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} Oct 30 17:57:29.516969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.516951 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"aa838616-f66c-424f-bbd8-eab8cc77ea9c","Type":"ContainerDied","Data":"2a860db12e701c67639750080d371c8387d75a67e4915cfa92c4fea7c2c293c5"} Oct 30 17:57:29.525455 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.525438 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.533394 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.533379 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.539814 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.539793 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.545945 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.545930 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.552589 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.552568 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.559226 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.559210 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.565674 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.565648 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.565909 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.565892 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.565956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.565920 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} err="failed to get container status \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" Oct 30 17:57:29.565956 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.565953 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.566192 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.566172 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.566253 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566202 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} err="failed to get container status \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" Oct 30 17:57:29.566253 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566224 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.566444 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.566426 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.566526 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566448 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} err="failed to get container status \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" Oct 30 17:57:29.566526 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566478 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.566679 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.566665 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.566757 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566682 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} err="failed to get container status \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" Oct 30 17:57:29.566757 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566692 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.566910 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.566894 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.566948 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566915 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} err="failed to get container status \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" Oct 30 17:57:29.566948 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.566931 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.567125 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.567106 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.567210 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.567125 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} err="failed to get container status \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" Oct 30 17:57:29.567210 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.567135 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.567709 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:29.567683 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.567709 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.567716 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} err="failed to get container status \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" Oct 30 17:57:29.567881 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.567736 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.568099 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568072 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} err="failed to get container status \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" Oct 30 17:57:29.568099 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568097 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.568366 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568341 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} err="failed to get container status \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" Oct 30 17:57:29.568450 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568369 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.568684 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568666 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} err="failed to get container status \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" Oct 30 17:57:29.568684 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568684 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.568900 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568881 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:29.568969 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568942 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} err="failed to get container status \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" Oct 30 17:57:29.569008 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.568965 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.569188 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569170 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} err="failed to get container status \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" Oct 30 17:57:29.569244 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569189 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.569408 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569390 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} err="failed to get container status \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" Oct 30 17:57:29.569449 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569409 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.569653 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569635 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} err="failed to get container status \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" Oct 30 17:57:29.569720 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569655 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.569871 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569855 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} err="failed to get container status \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" Oct 30 17:57:29.569918 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.569872 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.570080 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570064 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} err="failed to get container status \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" Oct 30 17:57:29.570118 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570079 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.570276 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570259 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} err="failed to get container status \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" Oct 30 17:57:29.570338 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570278 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.570506 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570490 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} err="failed to get container status \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" Oct 30 17:57:29.570559 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570507 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.570691 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570675 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} err="failed to get container status \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" Oct 30 17:57:29.570735 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570691 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.570866 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570851 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} err="failed to get container status \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" Oct 30 17:57:29.570906 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.570866 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.571054 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571036 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} err="failed to get container status \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" Oct 30 17:57:29.571092 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571054 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.571244 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571229 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} err="failed to get container status \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" Oct 30 17:57:29.571288 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571243 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.571425 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571410 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} err="failed to get container status \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" Oct 30 17:57:29.571485 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571424 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.571644 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571629 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} err="failed to get container status \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" Oct 30 17:57:29.571644 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571644 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.571843 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571826 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} err="failed to get container status \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" Oct 30 17:57:29.571906 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.571845 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.572053 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572038 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} err="failed to get container status \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" Oct 30 17:57:29.572101 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572052 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.572239 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572220 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} err="failed to get container status \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" Oct 30 17:57:29.572301 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572240 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.572514 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572497 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} err="failed to get container status \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" Oct 30 17:57:29.572560 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572515 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.572741 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572726 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} err="failed to get container status \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" Oct 30 17:57:29.572741 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572740 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.572983 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572963 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} err="failed to get container status \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" Oct 30 17:57:29.573030 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.572984 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.573196 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.573180 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} err="failed to get container status \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" Oct 30 17:57:29.573243 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.573195 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.573440 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.573420 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} err="failed to get container status \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" Oct 30 17:57:29.573440 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.573440 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.573823 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.573798 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} err="failed to get container status \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" Oct 30 17:57:29.573823 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.573823 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.574452 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.574432 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} err="failed to get container status \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" Oct 30 17:57:29.574452 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.574450 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.574614 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.574596 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:29.574777 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.574755 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} err="failed to get container status \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" Oct 30 17:57:29.574777 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.574776 6966 scope.go:122] "RemoveContainer" containerID="1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6" Oct 30 17:57:29.575033 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575016 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6"} err="failed to get container status \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": rpc error: code = NotFound desc = could not find container \"1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6\": container with ID starting with 1ffe29d176dbfb5e38fded10f965d24585af537fe7f8c91901b6e53ed9c6f6d6 not found: ID does not exist" Oct 30 17:57:29.575106 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575035 6966 scope.go:122] "RemoveContainer" containerID="35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2" Oct 30 17:57:29.575298 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575282 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2"} err="failed to get container status \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": rpc error: code = NotFound desc = could not find container \"35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2\": container with ID starting with 35b176e706adda0999b9b0e3319c91df874501066d4a7452ac34dbc353355ac2 not found: ID does not exist" Oct 30 17:57:29.575367 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575299 6966 scope.go:122] "RemoveContainer" containerID="f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964" Oct 30 17:57:29.575560 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575544 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964"} err="failed to get container status \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": rpc error: code = NotFound desc = could not find container \"f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964\": container with ID starting with f1016ea11d0387b4380949991b5ab93fd2f45feca264bc2fe510eac3d94aa964 not found: ID does not exist" Oct 30 17:57:29.575622 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575560 6966 scope.go:122] "RemoveContainer" containerID="7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8" Oct 30 17:57:29.575816 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575798 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8"} err="failed to get container status \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": rpc error: code = NotFound desc = could not find container \"7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8\": container with ID starting with 7fee1a4a84526bc9b76519b6dcbb53c898b5d5cb1fbcda56ef0ee2a77ca468e8 not found: ID does not exist" Oct 30 17:57:29.575859 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.575816 6966 scope.go:122] "RemoveContainer" containerID="25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4" Oct 30 17:57:29.576067 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.576048 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4"} err="failed to get container status \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": rpc error: code = NotFound desc = could not find container \"25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4\": container with ID starting with 25fb5dea352adc4d78bdeed456de41cbc717e75090665b00ddb6d4c99dcef8f4 not found: ID does not exist" Oct 30 17:57:29.576115 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.576068 6966 scope.go:122] "RemoveContainer" containerID="0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08" Oct 30 17:57:29.576302 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.576285 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08"} err="failed to get container status \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": rpc error: code = NotFound desc = could not find container \"0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08\": container with ID starting with 0335b0b05634d0e5c695134b40c8bc55eeb1eac41d1aea19992dd04a3696de08 not found: ID does not exist" Oct 30 17:57:29.576365 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.576303 6966 scope.go:122] "RemoveContainer" containerID="c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1" Oct 30 17:57:29.576587 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.576568 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1"} err="failed to get container status \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": rpc error: code = NotFound desc = could not find container \"c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1\": container with ID starting with c222667abf1788e2c828cf83006b8f0fffabe225aeb06997abac3179268443f1 not found: ID does not exist" Oct 30 17:57:29.602749 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602724 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602749 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602750 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602761 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56kz9\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-kube-api-access-56kz9\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602771 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602781 6966 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602791 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602799 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602806 6966 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-main-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602815 6966 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa838616-f66c-424f-bbd8-eab8cc77ea9c-metrics-client-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602824 6966 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602833 6966 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/aa838616-f66c-424f-bbd8-eab8cc77ea9c-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.602877 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.602842 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/aa838616-f66c-424f-bbd8-eab8cc77ea9c-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:29.606530 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606506 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:29.606892 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606875 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="config-reloader" Oct 30 17:57:29.606931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606893 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="config-reloader" Oct 30 17:57:29.606931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606909 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="prom-label-proxy" Oct 30 17:57:29.606931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606914 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="prom-label-proxy" Oct 30 17:57:29.606931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606923 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-metric" Oct 30 17:57:29.606931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606928 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-metric" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606944 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606948 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606957 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="init-config-reloader" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606963 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="init-config-reloader" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606972 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-web" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606976 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-web" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606983 6966 cpu_manager.go:414] "containerMap: removing container" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="alertmanager" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.606988 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="alertmanager" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607045 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-metric" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607056 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607062 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="init-config-reloader" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607068 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="kube-rbac-proxy-web" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607079 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="alertmanager" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607087 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="config-reloader" Oct 30 17:57:29.607139 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.607097 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" containerName="prom-label-proxy" Oct 30 17:57:29.648563 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.648544 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:29.648699 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.648683 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.667421 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667400 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 17:57:29.667541 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667403 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 17:57:29.667541 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667400 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 17:57:29.667541 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667524 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 17:57:29.667636 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667564 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 17:57:29.667636 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667576 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 17:57:29.667636 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667610 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 17:57:29.667718 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667686 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-t7pkc\"" Oct 30 17:57:29.667750 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.667724 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 17:57:29.673254 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.673226 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:29.679039 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.679019 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 17:57:29.699933 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.699912 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58cbf4966f-qjsxr"] Oct 30 17:57:29.704830 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:29.704797 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c7525d_92a7_400b_8367_b88234b4a79c.slice/crio-9992c9056060b518f1aa4d9fc716f587c556210deca7e4a340765ebf583bcac6 WatchSource:0}: Error finding container 9992c9056060b518f1aa4d9fc716f587c556210deca7e4a340765ebf583bcac6: Status 404 returned error can't find the container with id 9992c9056060b518f1aa4d9fc716f587c556210deca7e4a340765ebf583bcac6 Oct 30 17:57:29.803923 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.803863 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.803923 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.803901 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804055 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.803933 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804055 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804045 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804145 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804072 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-web-config\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804145 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804093 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-config-out\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804249 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804230 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804311 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804269 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804311 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804300 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jppl\" (UniqueName: \"kubernetes.io/projected/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-kube-api-access-2jppl\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804404 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804327 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804451 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804414 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804531 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804478 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.804579 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.804528 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.905699 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905672 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.905849 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905703 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.905849 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905722 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.905849 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905743 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.905849 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905760 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-web-config\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.905849 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905779 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-config-out\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906107 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905919 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906107 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905967 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906107 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.905989 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jppl\" (UniqueName: \"kubernetes.io/projected/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-kube-api-access-2jppl\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906107 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.906009 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906107 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.906044 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906107 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.906093 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906379 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.906124 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.906503 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.906488 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.907108 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.906983 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.908189 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.908158 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.908847 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.908821 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.908943 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.908903 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-config-out\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.909016 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.908987 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.909142 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.909107 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-web-config\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.909142 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.909131 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.909257 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.909183 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.909634 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.909613 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.910363 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.910344 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.910679 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.910657 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:29.930163 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:29.930144 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jppl\" (UniqueName: \"kubernetes.io/projected/b5b9edbf-0560-47ba-8c1b-aecca3a6365a-kube-api-access-2jppl\") pod \"alertmanager-main-0\" (UID: \"b5b9edbf-0560-47ba-8c1b-aecca3a6365a\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:30.071169 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.071144 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 17:57:30.414054 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.413907 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 17:57:30.419025 ip-10-0-136-241 kubenswrapper[6966]: W1030 17:57:30.419001 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5b9edbf_0560_47ba_8c1b_aecca3a6365a.slice/crio-82ba90824c15a0c49d552189d77d821a78908a0353aa52831d2575557c888baa WatchSource:0}: Error finding container 82ba90824c15a0c49d552189d77d821a78908a0353aa52831d2575557c888baa: Status 404 returned error can't find the container with id 82ba90824c15a0c49d552189d77d821a78908a0353aa52831d2575557c888baa Oct 30 17:57:30.521787 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.521764 6966 generic.go:356] "Generic (PLEG): container finished" podID="b5b9edbf-0560-47ba-8c1b-aecca3a6365a" containerID="efb62a64841ab1f7800a52fabe5911b08081290bb96114776b12d31fdc5428bf" exitCode=0 Oct 30 17:57:30.521890 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.521831 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerDied","Data":"efb62a64841ab1f7800a52fabe5911b08081290bb96114776b12d31fdc5428bf"} Oct 30 17:57:30.521890 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.521888 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"82ba90824c15a0c49d552189d77d821a78908a0353aa52831d2575557c888baa"} Oct 30 17:57:30.523340 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.523314 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbf4966f-qjsxr" event={"ID":"b8c7525d-92a7-400b-8367-b88234b4a79c","Type":"ContainerStarted","Data":"423efed800f216c1476c6c73c1a9962e731a0ca1c333b699de37dd28ace69fbc"} Oct 30 17:57:30.523434 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.523364 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58cbf4966f-qjsxr" event={"ID":"b8c7525d-92a7-400b-8367-b88234b4a79c","Type":"ContainerStarted","Data":"9992c9056060b518f1aa4d9fc716f587c556210deca7e4a340765ebf583bcac6"} Oct 30 17:57:30.617496 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:30.617439 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-58cbf4966f-qjsxr" podStartSLOduration=2.6174265930000002 podStartE2EDuration="2.617426593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:30.616784905 +0000 UTC m=+13.322970257" watchObservedRunningTime="2026-10-30 17:57:30.617426593 +0000 UTC m=+13.323611949" Oct 30 17:57:31.412957 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.412928 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa838616-f66c-424f-bbd8-eab8cc77ea9c" path="/var/lib/kubelet/pods/aa838616-f66c-424f-bbd8-eab8cc77ea9c/volumes" Oct 30 17:57:31.531806 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.531773 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"4ba0c083cc3069da1f44852182d10b8e5af1b712ae0a22218448f49cdaddbb66"} Oct 30 17:57:31.532202 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.531839 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"98d6f6f0cfffaf79b2548c86b6a9959a6fa5bcd219404481b27dd9600b810d03"} Oct 30 17:57:31.532202 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.531871 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"870f71b9f8074f4845984bc96008cb6c430bba159684a5841f7e423652cc0d3a"} Oct 30 17:57:31.532202 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.531928 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"f9095696e90853ea858a4b46ed43aed1762fd2fa81a3916ae73b69a9ca0b859f"} Oct 30 17:57:31.532202 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.531953 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"60541f248e58e075a8889afb0de841e19a5dec4f40ba91a7579518c7be5ebfca"} Oct 30 17:57:31.532202 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.531977 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5b9edbf-0560-47ba-8c1b-aecca3a6365a","Type":"ContainerStarted","Data":"e02853166c154c733366e33ebe58503748d835075c0f977a933da3b86422c69a"} Oct 30 17:57:31.594879 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:31.594835 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.589682404 podStartE2EDuration="2.594823003s" totalImagesPullingTime="0s" totalInitContainerRuntime="5.140599ms" isStatefulPod=false podCreationTimestamp="2026-10-30 17:57:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:57:31.593596669 +0000 UTC m=+14.299782029" watchObservedRunningTime="2026-10-30 17:57:31.594823003 +0000 UTC m=+14.301008351" Oct 30 17:57:32.226573 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:32.226541 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:32.432077 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:32.432043 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:35.757536 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:35.757493 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:35.959652 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:35.959621 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:39.374301 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:39.374265 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:39.374698 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:39.374358 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:39.379002 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:39.378980 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:39.565198 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:39.565174 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58cbf4966f-qjsxr" Oct 30 17:57:39.638623 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:39.638559 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9d75c56b-xtlgz"] Oct 30 17:57:43.440471 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.440399 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6ccccdd4d8-bz6n6" podUID="a2d95db0-3d14-46b9-99d5-902c2f740585" containerName="console" containerID="cri-o://0f95edba0d5169a6de26d1a5179e324e3a71ec10540e836d9a1055df00358607" gracePeriod=15 Oct 30 17:57:43.577047 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.577019 6966 generic.go:356] "Generic (PLEG): container finished" podID="a2d95db0-3d14-46b9-99d5-902c2f740585" containerID="0f95edba0d5169a6de26d1a5179e324e3a71ec10540e836d9a1055df00358607" exitCode=0 Oct 30 17:57:43.577184 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.577075 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccccdd4d8-bz6n6" event={"ID":"a2d95db0-3d14-46b9-99d5-902c2f740585","Type":"ContainerDied","Data":"0f95edba0d5169a6de26d1a5179e324e3a71ec10540e836d9a1055df00358607"} Oct 30 17:57:43.674852 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.674829 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:43.811630 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811601 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.811775 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811646 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.811775 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811676 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.811775 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811752 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.811943 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811792 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.811943 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811855 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.811943 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811907 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.812080 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.811939 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") pod \"a2d95db0-3d14-46b9-99d5-902c2f740585\" (UID: \"a2d95db0-3d14-46b9-99d5-902c2f740585\") " Oct 30 17:57:43.812155 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812127 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:43.812214 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812054 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:43.812214 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812058 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:43.812214 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812202 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:43.812369 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812263 6966 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-oauth-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.812369 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812283 6966 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-console-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.812369 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812299 6966 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2d95db0-3d14-46b9-99d5-902c2f740585-tmp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.812514 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.812498 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:43.814103 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.814075 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:43.814216 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.814145 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:43.814216 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.814196 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k" pod "a2d95db0-3d14-46b9-99d5-902c2f740585" (UID: "a2d95db0-3d14-46b9-99d5-902c2f740585"). InnerVolumeSpecName "kube-api-access-zdw5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:43.913602 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.913580 6966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-trusted-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.913602 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.913600 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdw5k\" (UniqueName: \"kubernetes.io/projected/a2d95db0-3d14-46b9-99d5-902c2f740585-kube-api-access-zdw5k\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.913731 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.913636 6966 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-oauth-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.913731 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.913648 6966 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2d95db0-3d14-46b9-99d5-902c2f740585-service-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:43.913731 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:43.913657 6966 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2d95db0-3d14-46b9-99d5-902c2f740585-console-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.495950 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.495918 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-fb8fd8fd8-5ks86" podUID="4652dca2-9b46-43e6-87d4-70c094ecf09a" containerName="console" containerID="cri-o://f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e" gracePeriod=15 Oct 30 17:57:44.582137 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.582116 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6ccccdd4d8-bz6n6" Oct 30 17:57:44.582137 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.582118 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6ccccdd4d8-bz6n6" event={"ID":"a2d95db0-3d14-46b9-99d5-902c2f740585","Type":"ContainerDied","Data":"7b263ef59b5544028eed7c2e19406c102d32c8050742013907ffa6acdf8b763e"} Oct 30 17:57:44.582268 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.582176 6966 scope.go:122] "RemoveContainer" containerID="0f95edba0d5169a6de26d1a5179e324e3a71ec10540e836d9a1055df00358607" Oct 30 17:57:44.630940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.630915 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6ccccdd4d8-bz6n6"] Oct 30 17:57:44.642297 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.642274 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6ccccdd4d8-bz6n6"] Oct 30 17:57:44.737722 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.737702 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:44.820511 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820482 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820624 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820610 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820711 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820695 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820768 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820738 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820830 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820772 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820830 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820816 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820936 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820850 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") pod \"4652dca2-9b46-43e6-87d4-70c094ecf09a\" (UID: \"4652dca2-9b46-43e6-87d4-70c094ecf09a\") " Oct 30 17:57:44.820991 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820933 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:44.821054 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.820936 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:57:44.821145 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.821123 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:44.821204 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.821145 6966 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4652dca2-9b46-43e6-87d4-70c094ecf09a-tmp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.821204 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.821164 6966 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-service-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.821337 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.821318 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:57:44.822648 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.822627 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:44.823071 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.823053 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:57:44.823113 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.823055 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2" pod "4652dca2-9b46-43e6-87d4-70c094ecf09a" (UID: "4652dca2-9b46-43e6-87d4-70c094ecf09a"). InnerVolumeSpecName "kube-api-access-khtd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:57:44.921526 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.921500 6966 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.921526 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.921524 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khtd2\" (UniqueName: \"kubernetes.io/projected/4652dca2-9b46-43e6-87d4-70c094ecf09a-kube-api-access-khtd2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.921671 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.921538 6966 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-oauth-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.921671 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.921550 6966 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/4652dca2-9b46-43e6-87d4-70c094ecf09a-oauth-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:44.921671 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:44.921562 6966 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/4652dca2-9b46-43e6-87d4-70c094ecf09a-console-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:57:45.045770 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.045743 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-pjfgc" Oct 30 17:57:45.410456 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.410414 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d95db0-3d14-46b9-99d5-902c2f740585" path="/var/lib/kubelet/pods/a2d95db0-3d14-46b9-99d5-902c2f740585/volumes" Oct 30 17:57:45.587500 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.587474 6966 generic.go:356] "Generic (PLEG): container finished" podID="4652dca2-9b46-43e6-87d4-70c094ecf09a" containerID="f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e" exitCode=0 Oct 30 17:57:45.587931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.587526 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-fb8fd8fd8-5ks86" Oct 30 17:57:45.587931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.587555 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb8fd8fd8-5ks86" event={"ID":"4652dca2-9b46-43e6-87d4-70c094ecf09a","Type":"ContainerDied","Data":"f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e"} Oct 30 17:57:45.587931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.587617 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-fb8fd8fd8-5ks86" event={"ID":"4652dca2-9b46-43e6-87d4-70c094ecf09a","Type":"ContainerDied","Data":"37eb554ab8065a9cb82433dae28fe8fb712100a7a169c254382f237b96ed9864"} Oct 30 17:57:45.587931 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.587650 6966 scope.go:122] "RemoveContainer" containerID="f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e" Oct 30 17:57:45.595441 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.595426 6966 scope.go:122] "RemoveContainer" containerID="f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e" Oct 30 17:57:45.595727 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:45.595711 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e\": container with ID starting with f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e not found: ID does not exist" containerID="f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e" Oct 30 17:57:45.595778 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.595734 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e"} err="failed to get container status \"f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e\": rpc error: code = NotFound desc = could not find container \"f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e\": container with ID starting with f67dd78d1d4e02ee70d996cce9cf59aab2f5759be40b2c74d07200ae85f8df2e not found: ID does not exist" Oct 30 17:57:45.635443 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.635423 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-fb8fd8fd8-5ks86"] Oct 30 17:57:45.646778 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:45.646761 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-fb8fd8fd8-5ks86"] Oct 30 17:57:47.198111 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:57:47.198077 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 17:57:47.410471 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:47.410432 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4652dca2-9b46-43e6-87d4-70c094ecf09a" path="/var/lib/kubelet/pods/4652dca2-9b46-43e6-87d4-70c094ecf09a/volumes" Oct 30 17:57:47.756359 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:47.756334 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-bzkm6" Oct 30 17:57:47.901378 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:47.901347 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:48.105940 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:48.105909 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:53.857109 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:53.857078 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:57:56.965426 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:57:56.965389 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:04.658029 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.657985 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-f9d75c56b-xtlgz" podUID="e23038e0-b513-49b0-bf52-f66a54d0c3d7" containerName="console" containerID="cri-o://f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e" gracePeriod=15 Oct 30 17:58:04.899444 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.899418 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:58:04.963699 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963639 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963699 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963686 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963886 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963709 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963886 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963756 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963886 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963779 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963886 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963800 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963886 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963843 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.963886 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.963877 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") pod \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\" (UID: \"e23038e0-b513-49b0-bf52-f66a54d0c3d7\") " Oct 30 17:58:04.964193 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.964139 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:04.964193 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.964150 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:04.964332 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.964287 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:04.964479 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.964366 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:58:04.964804 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.964434 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:58:04.966984 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.966840 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:04.969652 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.969288 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "kube-api-access-cfq5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:58:04.969652 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:04.969387 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert" pod "e23038e0-b513-49b0-bf52-f66a54d0c3d7" (UID: "e23038e0-b513-49b0-bf52-f66a54d0c3d7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:58:05.064685 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064661 6966 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064685 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064682 6966 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-oauth-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064797 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064693 6966 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-service-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064797 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064718 6966 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-oauth-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064797 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064728 6966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e23038e0-b513-49b0-bf52-f66a54d0c3d7-trusted-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064797 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064737 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfq5h\" (UniqueName: \"kubernetes.io/projected/e23038e0-b513-49b0-bf52-f66a54d0c3d7-kube-api-access-cfq5h\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064797 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064745 6966 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e23038e0-b513-49b0-bf52-f66a54d0c3d7-console-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.064797 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.064754 6966 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e23038e0-b513-49b0-bf52-f66a54d0c3d7-tmp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 17:58:05.663115 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.663081 6966 generic.go:356] "Generic (PLEG): container finished" podID="e23038e0-b513-49b0-bf52-f66a54d0c3d7" containerID="f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e" exitCode=0 Oct 30 17:58:05.663487 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.663148 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-f9d75c56b-xtlgz" Oct 30 17:58:05.663487 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.663172 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9d75c56b-xtlgz" event={"ID":"e23038e0-b513-49b0-bf52-f66a54d0c3d7","Type":"ContainerDied","Data":"f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e"} Oct 30 17:58:05.663487 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.663225 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-f9d75c56b-xtlgz" event={"ID":"e23038e0-b513-49b0-bf52-f66a54d0c3d7","Type":"ContainerDied","Data":"a53af0274c50215d975dcaf762287b0bd81e66c9ef251a1e919067e63e65adfd"} Oct 30 17:58:05.663487 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.663251 6966 scope.go:122] "RemoveContainer" containerID="f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e" Oct 30 17:58:05.671016 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.671000 6966 scope.go:122] "RemoveContainer" containerID="f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e" Oct 30 17:58:05.671279 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:58:05.671255 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e\": container with ID starting with f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e not found: ID does not exist" containerID="f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e" Oct 30 17:58:05.671334 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.671289 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e"} err="failed to get container status \"f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e\": rpc error: code = NotFound desc = could not find container \"f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e\": container with ID starting with f79779d3af987c99911a499dda84211a5eee1d0790c7f1f2bc7633743bb2f45e not found: ID does not exist" Oct 30 17:58:05.710791 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.710767 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-f9d75c56b-xtlgz"] Oct 30 17:58:05.721988 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:05.721967 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-f9d75c56b-xtlgz"] Oct 30 17:58:07.410294 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:07.410263 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e23038e0-b513-49b0-bf52-f66a54d0c3d7" path="/var/lib/kubelet/pods/e23038e0-b513-49b0-bf52-f66a54d0c3d7/volumes" Oct 30 17:58:16.497751 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:16.497711 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.198662 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:58:17.198625 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 17:58:17.383331 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:17.383300 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:17.592029 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:17.592000 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:33.001237 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:33.001191 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:33.211500 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:33.211452 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:58:47.200360 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:58:47.200324 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 17:58:59.673939 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:58:59.673859 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:16.663906 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:59:16.663867 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:17.199082 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:59:17.199048 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 17:59:25.379477 ip-10-0-136-241 kubenswrapper[6966]: I1030 17:59:25.379421 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 17:59:47.198993 ip-10-0-136-241 kubenswrapper[6966]: E1030 17:59:47.198957 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:00:16.756996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:00:16.756959 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:00:17.197878 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:00:17.197845 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:00:47.199866 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:00:47.199790 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:01:16.885694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:01:16.885652 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:17.200316 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:01:17.200232 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:01:30.381049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:01:30.381011 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:33.374886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:01:33.374845 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:33.586095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:01:33.586064 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:47.198970 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:01:47.198935 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:01:50.754199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:01:50.754167 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:01:50.965234 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:01:50.965197 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:17.038249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:02:17.038208 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:02:17.198369 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:02:17.198339 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:02:47.198270 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:02:47.198228 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:03:17.168646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:03:17.168604 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:03:17.198692 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:03:17.198662 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:03:47.198298 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:03:47.198219 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:04:17.198332 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:04:17.198295 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:04:17.277171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:04:17.277135 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:04:47.198918 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:04:47.198880 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:05:17.198998 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:05:17.198918 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:05:17.427765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:05:17.427732 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:05:47.198784 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:05:47.198748 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:06:08.631786 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:08.631752 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:08.865558 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:08.865520 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:09.469543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469514 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct"] Oct 30 18:06:09.469803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469791 6966 cpu_manager.go:414] "containerMap: removing container" podUID="4652dca2-9b46-43e6-87d4-70c094ecf09a" containerName="console" Oct 30 18:06:09.469850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469805 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4652dca2-9b46-43e6-87d4-70c094ecf09a" containerName="console" Oct 30 18:06:09.469850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469816 6966 cpu_manager.go:414] "containerMap: removing container" podUID="e23038e0-b513-49b0-bf52-f66a54d0c3d7" containerName="console" Oct 30 18:06:09.469850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469821 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e23038e0-b513-49b0-bf52-f66a54d0c3d7" containerName="console" Oct 30 18:06:09.469850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469839 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a2d95db0-3d14-46b9-99d5-902c2f740585" containerName="console" Oct 30 18:06:09.469850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469844 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d95db0-3d14-46b9-99d5-902c2f740585" containerName="console" Oct 30 18:06:09.469995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469906 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d95db0-3d14-46b9-99d5-902c2f740585" containerName="console" Oct 30 18:06:09.469995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469917 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="4652dca2-9b46-43e6-87d4-70c094ecf09a" containerName="console" Oct 30 18:06:09.469995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.469923 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e23038e0-b513-49b0-bf52-f66a54d0c3d7" containerName="console" Oct 30 18:06:09.472828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.472809 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.487279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.487258 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:06:09.498735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.498719 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:06:09.498842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.498719 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-b89w5\"" Oct 30 18:06:09.509494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.509450 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct"] Oct 30 18:06:09.577407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.577380 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.577545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.577418 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.577590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.577548 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpcd\" (UniqueName: \"kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.678840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.678818 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpcd\" (UniqueName: \"kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.679154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.678852 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.679154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.678877 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.679223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.679170 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.679223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.679206 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.701974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.701949 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpcd\" (UniqueName: \"kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:09.788082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:09.788034 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:10.082000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:10.081974 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct"] Oct 30 18:06:10.087028 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:06:10.087002 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc16e3ffb_2e2c_42e5_aaf1_701fe364e6e6.slice/crio-759bf3ea7f0bd1f4af6aac962998dfbd422bcfdc21ace745456b1f9b7008c875 WatchSource:0}: Error finding container 759bf3ea7f0bd1f4af6aac962998dfbd422bcfdc21ace745456b1f9b7008c875: Status 404 returned error can't find the container with id 759bf3ea7f0bd1f4af6aac962998dfbd422bcfdc21ace745456b1f9b7008c875 Oct 30 18:06:10.089016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:10.088999 6966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:06:10.323054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:10.323025 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" event={"ID":"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6","Type":"ContainerStarted","Data":"759bf3ea7f0bd1f4af6aac962998dfbd422bcfdc21ace745456b1f9b7008c875"} Oct 30 18:06:15.344824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:15.344788 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" event={"ID":"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6","Type":"ContainerStarted","Data":"ebe3130bfceea872b0ae2e2a7b86d73c0a3a9c4270e5f283be70e64bf892b90a"} Oct 30 18:06:16.349307 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:16.349268 6966 generic.go:356] "Generic (PLEG): container finished" podID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerID="ebe3130bfceea872b0ae2e2a7b86d73c0a3a9c4270e5f283be70e64bf892b90a" exitCode=0 Oct 30 18:06:16.349744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:16.349342 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" event={"ID":"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6","Type":"ContainerDied","Data":"ebe3130bfceea872b0ae2e2a7b86d73c0a3a9c4270e5f283be70e64bf892b90a"} Oct 30 18:06:17.197951 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:06:17.197912 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:06:17.524990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:17.524878 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:19.362566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:19.362532 6966 generic.go:356] "Generic (PLEG): container finished" podID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerID="a7ddea56c0b6f928e70a8f74e915da79fa09a2dfb382cb3212f79a289a371f37" exitCode=0 Oct 30 18:06:19.362949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:19.362584 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" event={"ID":"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6","Type":"ContainerDied","Data":"a7ddea56c0b6f928e70a8f74e915da79fa09a2dfb382cb3212f79a289a371f37"} Oct 30 18:06:25.389037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:25.389006 6966 generic.go:356] "Generic (PLEG): container finished" podID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerID="a41fb8899365513ecf56c36992749774d4fc7888664b677b1c299668e9ac2250" exitCode=0 Oct 30 18:06:25.389431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:25.389087 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" event={"ID":"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6","Type":"ContainerDied","Data":"a41fb8899365513ecf56c36992749774d4fc7888664b677b1c299668e9ac2250"} Oct 30 18:06:26.510121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.510097 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:26.619134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.619104 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd\" (UniqueName: \"kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd\") pod \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " Oct 30 18:06:26.619262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.619159 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util\") pod \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " Oct 30 18:06:26.619262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.619197 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle\") pod \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\" (UID: \"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6\") " Oct 30 18:06:26.619822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.619795 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle" pod "c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" (UID: "c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:26.621280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.621253 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd" pod "c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" (UID: "c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6"). InnerVolumeSpecName "kube-api-access-qjpcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:06:26.623536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.623517 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util" pod "c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" (UID: "c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:06:26.719891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.719840 6966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:06:26.719891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.719863 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjpcd\" (UniqueName: \"kubernetes.io/projected/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-kube-api-access-qjpcd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:06:26.719891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:26.719873 6966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6-util\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:06:27.398278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:27.398256 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" Oct 30 18:06:27.398278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:27.398260 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cwf8ct" event={"ID":"c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6","Type":"ContainerDied","Data":"759bf3ea7f0bd1f4af6aac962998dfbd422bcfdc21ace745456b1f9b7008c875"} Oct 30 18:06:27.398492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:27.398302 6966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="759bf3ea7f0bd1f4af6aac962998dfbd422bcfdc21ace745456b1f9b7008c875" Oct 30 18:06:38.985632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:38.985550 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:40.434665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:40.434628 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:41.232111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:41.232074 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:42.998048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:42.998009 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:43.211170 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211138 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v"] Oct 30 18:06:43.211580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211561 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="util" Oct 30 18:06:43.211580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211580 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="util" Oct 30 18:06:43.211722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211592 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="pull" Oct 30 18:06:43.211722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211597 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="pull" Oct 30 18:06:43.211722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211608 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="extract" Oct 30 18:06:43.211722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211614 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="extract" Oct 30 18:06:43.211722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.211669 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c16e3ffb-2e2c-42e5-aaf1-701fe364e6e6" containerName="extract" Oct 30 18:06:43.213668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.213649 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.229491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.229453 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 18:06:43.239955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.239935 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-zdfd6\"" Oct 30 18:06:43.240077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.240015 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:06:43.240863 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.240841 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v"] Oct 30 18:06:43.336647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.336623 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq25x\" (UniqueName: \"kubernetes.io/projected/88c52b88-6d78-463d-a7c8-1d43ab651bc8-kube-api-access-gq25x\") pod \"cert-manager-operator-controller-manager-65fc875df4-kvx9v\" (UID: \"88c52b88-6d78-463d-a7c8-1d43ab651bc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.336781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.336667 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88c52b88-6d78-463d-a7c8-1d43ab651bc8-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-kvx9v\" (UID: \"88c52b88-6d78-463d-a7c8-1d43ab651bc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.437293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.437266 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq25x\" (UniqueName: \"kubernetes.io/projected/88c52b88-6d78-463d-a7c8-1d43ab651bc8-kube-api-access-gq25x\") pod \"cert-manager-operator-controller-manager-65fc875df4-kvx9v\" (UID: \"88c52b88-6d78-463d-a7c8-1d43ab651bc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.437417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.437309 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88c52b88-6d78-463d-a7c8-1d43ab651bc8-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-kvx9v\" (UID: \"88c52b88-6d78-463d-a7c8-1d43ab651bc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.437716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.437699 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88c52b88-6d78-463d-a7c8-1d43ab651bc8-tmp\") pod \"cert-manager-operator-controller-manager-65fc875df4-kvx9v\" (UID: \"88c52b88-6d78-463d-a7c8-1d43ab651bc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.462049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.462021 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq25x\" (UniqueName: \"kubernetes.io/projected/88c52b88-6d78-463d-a7c8-1d43ab651bc8-kube-api-access-gq25x\") pod \"cert-manager-operator-controller-manager-65fc875df4-kvx9v\" (UID: \"88c52b88-6d78-463d-a7c8-1d43ab651bc8\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.524008 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.523979 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" Oct 30 18:06:43.848793 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:43.848762 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v"] Oct 30 18:06:43.854903 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:06:43.854870 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c52b88_6d78_463d_a7c8_1d43ab651bc8.slice/crio-d2b56575b69898162ead2bdbe9b2378996cfe0b5671032a6dc10bffae59b1209 WatchSource:0}: Error finding container d2b56575b69898162ead2bdbe9b2378996cfe0b5671032a6dc10bffae59b1209: Status 404 returned error can't find the container with id d2b56575b69898162ead2bdbe9b2378996cfe0b5671032a6dc10bffae59b1209 Oct 30 18:06:44.457358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:44.457316 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" event={"ID":"88c52b88-6d78-463d-a7c8-1d43ab651bc8","Type":"ContainerStarted","Data":"d2b56575b69898162ead2bdbe9b2378996cfe0b5671032a6dc10bffae59b1209"} Oct 30 18:06:44.669587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:44.669552 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:44.952035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:44.951999 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:45.471764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:45.471583 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" event={"ID":"88c52b88-6d78-463d-a7c8-1d43ab651bc8","Type":"ContainerStarted","Data":"4938a716f2a628137ca5f39616080ba1ec206cdba790b3083937095e2e67fc55"} Oct 30 18:06:45.537166 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:45.537109 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-65fc875df4-kvx9v" podStartSLOduration=1.225276809 podStartE2EDuration="2.53709543s" totalImagesPullingTime="1.311818621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:45.525549833 +0000 UTC m=+568.231735193" watchObservedRunningTime="2026-10-30 18:06:45.53709543 +0000 UTC m=+568.243280784" Oct 30 18:06:46.368798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:46.368745 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:46.600097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:46.600052 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:47.198170 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:06:47.198137 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:06:47.438366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.438338 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl"] Oct 30 18:06:47.440563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.440540 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.465099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.465052 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:06:47.465099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.465066 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:06:47.465099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.465073 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ctgtj\"" Oct 30 18:06:47.465812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.465795 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl"] Oct 30 18:06:47.569284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.569255 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28102cd8-02f0-48b9-a779-4ae1a77cfe43-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2jpnl\" (UID: \"28102cd8-02f0-48b9-a779-4ae1a77cfe43\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.569451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.569316 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vh25\" (UniqueName: \"kubernetes.io/projected/28102cd8-02f0-48b9-a779-4ae1a77cfe43-kube-api-access-6vh25\") pod \"cert-manager-webhook-5dbbbf75cc-2jpnl\" (UID: \"28102cd8-02f0-48b9-a779-4ae1a77cfe43\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.669952 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.669917 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28102cd8-02f0-48b9-a779-4ae1a77cfe43-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2jpnl\" (UID: \"28102cd8-02f0-48b9-a779-4ae1a77cfe43\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.670307 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.669969 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vh25\" (UniqueName: \"kubernetes.io/projected/28102cd8-02f0-48b9-a779-4ae1a77cfe43-kube-api-access-6vh25\") pod \"cert-manager-webhook-5dbbbf75cc-2jpnl\" (UID: \"28102cd8-02f0-48b9-a779-4ae1a77cfe43\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.692413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.692386 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/28102cd8-02f0-48b9-a779-4ae1a77cfe43-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-2jpnl\" (UID: \"28102cd8-02f0-48b9-a779-4ae1a77cfe43\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.692554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.692541 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vh25\" (UniqueName: \"kubernetes.io/projected/28102cd8-02f0-48b9-a779-4ae1a77cfe43-kube-api-access-6vh25\") pod \"cert-manager-webhook-5dbbbf75cc-2jpnl\" (UID: \"28102cd8-02f0-48b9-a779-4ae1a77cfe43\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:47.758538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:47.758485 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:48.050984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:48.050907 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl"] Oct 30 18:06:48.056985 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:06:48.056961 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28102cd8_02f0_48b9_a779_4ae1a77cfe43.slice/crio-3d36b449cca67203621f055f4a17746bc42a930cf98158f646e8d73ebc91116d WatchSource:0}: Error finding container 3d36b449cca67203621f055f4a17746bc42a930cf98158f646e8d73ebc91116d: Status 404 returned error can't find the container with id 3d36b449cca67203621f055f4a17746bc42a930cf98158f646e8d73ebc91116d Oct 30 18:06:48.486055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:48.486023 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" event={"ID":"28102cd8-02f0-48b9-a779-4ae1a77cfe43","Type":"ContainerStarted","Data":"3d36b449cca67203621f055f4a17746bc42a930cf98158f646e8d73ebc91116d"} Oct 30 18:06:48.811345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:48.811309 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:49.045344 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:49.045284 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:49.246584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:49.246497 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:49.455903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:49.455869 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:51.506814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:51.506774 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" event={"ID":"28102cd8-02f0-48b9-a779-4ae1a77cfe43","Type":"ContainerStarted","Data":"9c6e94d8bae3bae91fafc6f4a608c0316e545ac51550b6ab99ab0da738637676"} Oct 30 18:06:51.507304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:51.506917 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:51.545752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:51.545694 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" podStartSLOduration=2.103063962 podStartE2EDuration="4.545678111s" totalImagesPullingTime="2.442614149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:51.544842811 +0000 UTC m=+574.251028184" watchObservedRunningTime="2026-10-30 18:06:51.545678111 +0000 UTC m=+574.251863469" Oct 30 18:06:56.073631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.073598 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:06:56.650840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.650812 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-q6q54"] Oct 30 18:06:56.671914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.671892 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.674141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.674117 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-q6q54"] Oct 30 18:06:56.696440 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.696423 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-b9rnp\"" Oct 30 18:06:56.836734 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.836706 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d84e0480-3709-4f21-a683-e30f3b70b3c7-bound-sa-token\") pod \"cert-manager-597b95cb84-q6q54\" (UID: \"d84e0480-3709-4f21-a683-e30f3b70b3c7\") " pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.836829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.836741 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzckh\" (UniqueName: \"kubernetes.io/projected/d84e0480-3709-4f21-a683-e30f3b70b3c7-kube-api-access-gzckh\") pod \"cert-manager-597b95cb84-q6q54\" (UID: \"d84e0480-3709-4f21-a683-e30f3b70b3c7\") " pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.937986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.937922 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d84e0480-3709-4f21-a683-e30f3b70b3c7-bound-sa-token\") pod \"cert-manager-597b95cb84-q6q54\" (UID: \"d84e0480-3709-4f21-a683-e30f3b70b3c7\") " pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.937986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.937956 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzckh\" (UniqueName: \"kubernetes.io/projected/d84e0480-3709-4f21-a683-e30f3b70b3c7-kube-api-access-gzckh\") pod \"cert-manager-597b95cb84-q6q54\" (UID: \"d84e0480-3709-4f21-a683-e30f3b70b3c7\") " pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.960316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.960288 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d84e0480-3709-4f21-a683-e30f3b70b3c7-bound-sa-token\") pod \"cert-manager-597b95cb84-q6q54\" (UID: \"d84e0480-3709-4f21-a683-e30f3b70b3c7\") " pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.960428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.960383 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzckh\" (UniqueName: \"kubernetes.io/projected/d84e0480-3709-4f21-a683-e30f3b70b3c7-kube-api-access-gzckh\") pod \"cert-manager-597b95cb84-q6q54\" (UID: \"d84e0480-3709-4f21-a683-e30f3b70b3c7\") " pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:56.996763 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:56.996741 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-q6q54" Oct 30 18:06:57.307871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:57.307801 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-q6q54"] Oct 30 18:06:57.313751 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:06:57.313720 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd84e0480_3709_4f21_a683_e30f3b70b3c7.slice/crio-df61ffa531a44cd995eb1022ba1baa2aff9fbf4b2658f938e326b377af99b13e WatchSource:0}: Error finding container df61ffa531a44cd995eb1022ba1baa2aff9fbf4b2658f938e326b377af99b13e: Status 404 returned error can't find the container with id df61ffa531a44cd995eb1022ba1baa2aff9fbf4b2658f938e326b377af99b13e Oct 30 18:06:57.513112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:57.513084 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-2jpnl" Oct 30 18:06:57.532925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:57.532892 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-q6q54" event={"ID":"d84e0480-3709-4f21-a683-e30f3b70b3c7","Type":"ContainerStarted","Data":"32dffc56af67739ec39c156fb14bcdf77649b723930ffab9e9da2a1ec44c8d91"} Oct 30 18:06:57.533050 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:57.532929 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-q6q54" event={"ID":"d84e0480-3709-4f21-a683-e30f3b70b3c7","Type":"ContainerStarted","Data":"df61ffa531a44cd995eb1022ba1baa2aff9fbf4b2658f938e326b377af99b13e"} Oct 30 18:06:57.591722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:06:57.591671 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-q6q54" podStartSLOduration=1.591650558 podStartE2EDuration="1.591650558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:57.591306197 +0000 UTC m=+580.297491554" watchObservedRunningTime="2026-10-30 18:06:57.591650558 +0000 UTC m=+580.297835914" Oct 30 18:07:17.199882 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:07:17.199844 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:07:17.636641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:17.636601 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:19.723874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:19.723841 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:19.937552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:19.937517 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:23.753247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.753214 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:23.789577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.789545 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-observability-system/odh-observability-5999d9c564-mm8f7"] Oct 30 18:07:23.793168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.793138 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.817774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.817748 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-webhook-cert\"" Oct 30 18:07:23.817898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.817750 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"openshift-service-ca.crt\"" Oct 30 18:07:23.817898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.817756 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-observability-system\"/\"kube-root-ca.crt\"" Oct 30 18:07:23.817898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.817792 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-dockercfg-pkwhq\"" Oct 30 18:07:23.817898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.817874 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-observability-system\"/\"odh-observability-metrics-tls\"" Oct 30 18:07:23.818701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.818682 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-5999d9c564-mm8f7"] Oct 30 18:07:23.835715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.835690 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-webhook-certs\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.835844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.835724 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-metrics-tls\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.835910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.835879 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5pwh\" (UniqueName: \"kubernetes.io/projected/eccd2c86-b51e-482f-9116-ddecf6a1b930-kube-api-access-r5pwh\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.936164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.936137 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-webhook-certs\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.936273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.936175 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-metrics-tls\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.936273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.936218 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5pwh\" (UniqueName: \"kubernetes.io/projected/eccd2c86-b51e-482f-9116-ddecf6a1b930-kube-api-access-r5pwh\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.936348 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:07:23.936287 6966 secret.go:189] Couldn't get secret odh-observability-system/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 18:07:23.936391 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:07:23.936379 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-webhook-certs podName:eccd2c86-b51e-482f-9116-ddecf6a1b930 nodeName:}" failed. No retries permitted until 2026-10-30 18:07:24.436350972 +0000 UTC m=+607.142536323 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-webhook-certs") pod "odh-observability-5999d9c564-mm8f7" (UID: "eccd2c86-b51e-482f-9116-ddecf6a1b930") : secret "odh-observability-webhook-cert" not found Oct 30 18:07:23.938609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.938586 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-metrics-tls\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:23.958484 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.958450 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:23.959324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:23.959304 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5pwh\" (UniqueName: \"kubernetes.io/projected/eccd2c86-b51e-482f-9116-ddecf6a1b930-kube-api-access-r5pwh\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:24.166205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:24.166178 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:24.439706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:24.439641 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-webhook-certs\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:24.441947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:24.441929 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/eccd2c86-b51e-482f-9116-ddecf6a1b930-webhook-certs\") pod \"odh-observability-5999d9c564-mm8f7\" (UID: \"eccd2c86-b51e-482f-9116-ddecf6a1b930\") " pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:24.514969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:24.514932 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:07:24.705919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:24.705848 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:25.000494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:25.000408 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-observability-system/odh-observability-5999d9c564-mm8f7"] Oct 30 18:07:25.005917 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:07:25.005877 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeccd2c86_b51e_482f_9116_ddecf6a1b930.slice/crio-f940f4ecc664e57e3295b22519a2931ad1ab3b520e31c40ebffc0c4df8fc1be7 WatchSource:0}: Error finding container f940f4ecc664e57e3295b22519a2931ad1ab3b520e31c40ebffc0c4df8fc1be7: Status 404 returned error can't find the container with id f940f4ecc664e57e3295b22519a2931ad1ab3b520e31c40ebffc0c4df8fc1be7 Oct 30 18:07:25.648064 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:25.647983 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" event={"ID":"eccd2c86-b51e-482f-9116-ddecf6a1b930","Type":"ContainerStarted","Data":"f940f4ecc664e57e3295b22519a2931ad1ab3b520e31c40ebffc0c4df8fc1be7"} Oct 30 18:07:28.664224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:28.664190 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" event={"ID":"eccd2c86-b51e-482f-9116-ddecf6a1b930","Type":"ContainerStarted","Data":"e30dc48b1e71f304cf89ed947b7f7ede4a7073dfbc9f0af43c57baefac002222"} Oct 30 18:07:28.664645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:28.664407 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:28.704564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:28.704523 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" podStartSLOduration=3.07283276 podStartE2EDuration="5.704512107s" totalImagesPullingTime="2.631679347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:28.703588147 +0000 UTC m=+611.409773503" watchObservedRunningTime="2026-10-30 18:07:28.704512107 +0000 UTC m=+611.410697483" Oct 30 18:07:39.670077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:07:39.670042 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:07:47.198975 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:07:47.198937 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:08:17.198471 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:08:17.198382 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:08:17.768300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:17.768266 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:47.198635 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:08:47.198600 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:08:48.539609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:48.539568 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:48.771925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:48.771891 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:49.033404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.033368 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph"] Oct 30 18:08:49.036706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.036684 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.061892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.061866 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:08:49.062039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.061940 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e1df189bb31fb8c419c1e848482afda4dccb51-08dd003f-dockercfg-qxr9z\"" Oct 30 18:08:49.062039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.062004 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:08:49.062862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.062839 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph"] Oct 30 18:08:49.141313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.141285 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cc8z\" (UniqueName: \"kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.141428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.141332 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.141428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.141415 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.241927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.241899 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.242049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.241951 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.242049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.241990 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cc8z\" (UniqueName: \"kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.242255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.242235 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.242295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.242271 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.265046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.265024 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cc8z\" (UniqueName: \"kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z\") pod \"e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.351751 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.351729 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:49.447721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.447688 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm"] Oct 30 18:08:49.451408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.451388 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.465381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.465361 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6fb1a13d619f6ee73e221b33f4898cc831d48b-0ad9862f-dockercfg-bvjvg\"" Oct 30 18:08:49.478660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.478639 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm"] Oct 30 18:08:49.644907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.644834 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzftd\" (UniqueName: \"kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.644907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.644876 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.645321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.644970 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.647061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.647038 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph"] Oct 30 18:08:49.652710 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:08:49.652684 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod513bcca5_acad_46e5_ab3e_9752c22cc0da.slice/crio-b578a043b76009676e6a33db020273a817640bf5891aec6f8f2b46dd0916508b WatchSource:0}: Error finding container b578a043b76009676e6a33db020273a817640bf5891aec6f8f2b46dd0916508b: Status 404 returned error can't find the container with id b578a043b76009676e6a33db020273a817640bf5891aec6f8f2b46dd0916508b Oct 30 18:08:49.746102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.746071 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzftd\" (UniqueName: \"kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.746244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.746123 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.746244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.746181 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.746522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.746503 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.746604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.746552 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.769038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.769013 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzftd\" (UniqueName: \"kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd\") pod \"6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:49.989139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.989071 6966 generic.go:356] "Generic (PLEG): container finished" podID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerID="a8269dbd805d09f45eb2180bef308ed686d49e6430a5314592c701dfa20a5af1" exitCode=0 Oct 30 18:08:49.989139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.989136 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" event={"ID":"513bcca5-acad-46e5-ab3e-9752c22cc0da","Type":"ContainerDied","Data":"a8269dbd805d09f45eb2180bef308ed686d49e6430a5314592c701dfa20a5af1"} Oct 30 18:08:49.989270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:49.989175 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" event={"ID":"513bcca5-acad-46e5-ab3e-9752c22cc0da","Type":"ContainerStarted","Data":"b578a043b76009676e6a33db020273a817640bf5891aec6f8f2b46dd0916508b"} Oct 30 18:08:50.067014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:50.066993 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:50.360759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:50.360731 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm"] Oct 30 18:08:50.365763 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:08:50.365735 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4be9e88_0385_4c8b_a2fb_c48fd1403898.slice/crio-a769d799b645108510c271712f53b6c4b13917d605dd5b6e7ecae341cafeeb9d WatchSource:0}: Error finding container a769d799b645108510c271712f53b6c4b13917d605dd5b6e7ecae341cafeeb9d: Status 404 returned error can't find the container with id a769d799b645108510c271712f53b6c4b13917d605dd5b6e7ecae341cafeeb9d Oct 30 18:08:50.995110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:50.995087 6966 generic.go:356] "Generic (PLEG): container finished" podID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerID="2d2672173e2e102ea31db97e1e5db91e849acb096bf39d77f3ddb1a65895b8f8" exitCode=0 Oct 30 18:08:50.995390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:50.995158 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" event={"ID":"a4be9e88-0385-4c8b-a2fb-c48fd1403898","Type":"ContainerDied","Data":"2d2672173e2e102ea31db97e1e5db91e849acb096bf39d77f3ddb1a65895b8f8"} Oct 30 18:08:50.995390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:50.995191 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" event={"ID":"a4be9e88-0385-4c8b-a2fb-c48fd1403898","Type":"ContainerStarted","Data":"a769d799b645108510c271712f53b6c4b13917d605dd5b6e7ecae341cafeeb9d"} Oct 30 18:08:52.001041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.001008 6966 generic.go:356] "Generic (PLEG): container finished" podID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerID="c87dae145fb16b7d929c39552ca08e825d2443dec13279939493d86968850e0e" exitCode=0 Oct 30 18:08:52.001415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.001086 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" event={"ID":"513bcca5-acad-46e5-ab3e-9752c22cc0da","Type":"ContainerDied","Data":"c87dae145fb16b7d929c39552ca08e825d2443dec13279939493d86968850e0e"} Oct 30 18:08:52.003083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.003063 6966 generic.go:356] "Generic (PLEG): container finished" podID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerID="4b59f728c3e828a1ae550ee3ad862edc19985754be49d42fde4dff440dcffbfa" exitCode=0 Oct 30 18:08:52.003167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.003120 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" event={"ID":"a4be9e88-0385-4c8b-a2fb-c48fd1403898","Type":"ContainerDied","Data":"4b59f728c3e828a1ae550ee3ad862edc19985754be49d42fde4dff440dcffbfa"} Oct 30 18:08:52.096616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.096532 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz"] Oct 30 18:08:52.100823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.100802 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.117182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.117160 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"07f0b9f07fd02dc1509b226b57404b9c198c3c-8adea440-dockercfg-22bjt\"" Oct 30 18:08:52.120973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.120951 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz"] Oct 30 18:08:52.163164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.163138 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vfqc\" (UniqueName: \"kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.163321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.163180 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.163321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.163201 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.264362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.264332 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.264362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.264364 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.264604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.264434 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vfqc\" (UniqueName: \"kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.264784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.264763 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.264852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.264784 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.287070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.287049 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vfqc\" (UniqueName: \"kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc\") pod \"07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.416355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.416288 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:52.714738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:52.714677 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz"] Oct 30 18:08:52.719494 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:08:52.719457 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb92d224_a3ba_4114_87e2_bf7135008b3a.slice/crio-1561752cb7094ce765e123bdc62491348977ec3a7054b2d562d91e2931b32b12 WatchSource:0}: Error finding container 1561752cb7094ce765e123bdc62491348977ec3a7054b2d562d91e2931b32b12: Status 404 returned error can't find the container with id 1561752cb7094ce765e123bdc62491348977ec3a7054b2d562d91e2931b32b12 Oct 30 18:08:53.009132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.009070 6966 generic.go:356] "Generic (PLEG): container finished" podID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerID="784df86e2299de9672b2d49f907a72ea05cdfbc8ed2a20454b4e79e285ca78d1" exitCode=0 Oct 30 18:08:53.009534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.009160 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" event={"ID":"a4be9e88-0385-4c8b-a2fb-c48fd1403898","Type":"ContainerDied","Data":"784df86e2299de9672b2d49f907a72ea05cdfbc8ed2a20454b4e79e285ca78d1"} Oct 30 18:08:53.011262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.011242 6966 generic.go:356] "Generic (PLEG): container finished" podID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerID="01d2bfbfefe6b09262d75560d74752c14c2c6a4cbc2a30bb77f7088a77266c66" exitCode=0 Oct 30 18:08:53.011370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.011304 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" event={"ID":"513bcca5-acad-46e5-ab3e-9752c22cc0da","Type":"ContainerDied","Data":"01d2bfbfefe6b09262d75560d74752c14c2c6a4cbc2a30bb77f7088a77266c66"} Oct 30 18:08:53.012614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.012592 6966 generic.go:356] "Generic (PLEG): container finished" podID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerID="866840dd3b97fe8a35f159efe98c4fda11d1daf53881cde75ca85983f94367fe" exitCode=0 Oct 30 18:08:53.012697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.012636 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" event={"ID":"fb92d224-a3ba-4114-87e2-bf7135008b3a","Type":"ContainerDied","Data":"866840dd3b97fe8a35f159efe98c4fda11d1daf53881cde75ca85983f94367fe"} Oct 30 18:08:53.012697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:53.012663 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" event={"ID":"fb92d224-a3ba-4114-87e2-bf7135008b3a","Type":"ContainerStarted","Data":"1561752cb7094ce765e123bdc62491348977ec3a7054b2d562d91e2931b32b12"} Oct 30 18:08:54.151366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.151343 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:54.176797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.176772 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:54.179056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179036 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z\" (UniqueName: \"kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z\") pod \"513bcca5-acad-46e5-ab3e-9752c22cc0da\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " Oct 30 18:08:54.179157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179102 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util\") pod \"513bcca5-acad-46e5-ab3e-9752c22cc0da\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " Oct 30 18:08:54.179157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179144 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util\") pod \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " Oct 30 18:08:54.179271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179218 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle\") pod \"513bcca5-acad-46e5-ab3e-9752c22cc0da\" (UID: \"513bcca5-acad-46e5-ab3e-9752c22cc0da\") " Oct 30 18:08:54.179327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179275 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle\") pod \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " Oct 30 18:08:54.179327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179305 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd\" (UniqueName: \"kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd\") pod \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\" (UID: \"a4be9e88-0385-4c8b-a2fb-c48fd1403898\") " Oct 30 18:08:54.179955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.179904 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle" pod "513bcca5-acad-46e5-ab3e-9752c22cc0da" (UID: "513bcca5-acad-46e5-ab3e-9752c22cc0da"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:54.181150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.181122 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle" pod "a4be9e88-0385-4c8b-a2fb-c48fd1403898" (UID: "a4be9e88-0385-4c8b-a2fb-c48fd1403898"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:54.181298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.181274 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd" pod "a4be9e88-0385-4c8b-a2fb-c48fd1403898" (UID: "a4be9e88-0385-4c8b-a2fb-c48fd1403898"). InnerVolumeSpecName "kube-api-access-vzftd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:54.181433 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.181416 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z" pod "513bcca5-acad-46e5-ab3e-9752c22cc0da" (UID: "513bcca5-acad-46e5-ab3e-9752c22cc0da"). InnerVolumeSpecName "kube-api-access-4cc8z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:54.185571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.185548 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util" pod "513bcca5-acad-46e5-ab3e-9752c22cc0da" (UID: "513bcca5-acad-46e5-ab3e-9752c22cc0da"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:54.185641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.185621 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util" pod "a4be9e88-0385-4c8b-a2fb-c48fd1403898" (UID: "a4be9e88-0385-4c8b-a2fb-c48fd1403898"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:54.280408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.280381 6966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-util\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.280408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.280406 6966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-util\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.280587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.280416 6966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/513bcca5-acad-46e5-ab3e-9752c22cc0da-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.280587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.280424 6966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a4be9e88-0385-4c8b-a2fb-c48fd1403898-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.280587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.280433 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzftd\" (UniqueName: \"kubernetes.io/projected/a4be9e88-0385-4c8b-a2fb-c48fd1403898-kube-api-access-vzftd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:54.280587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:54.280442 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cc8z\" (UniqueName: \"kubernetes.io/projected/513bcca5-acad-46e5-ab3e-9752c22cc0da-kube-api-access-4cc8z\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:55.021851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.021818 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" event={"ID":"513bcca5-acad-46e5-ab3e-9752c22cc0da","Type":"ContainerDied","Data":"b578a043b76009676e6a33db020273a817640bf5891aec6f8f2b46dd0916508b"} Oct 30 18:08:55.022019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.021863 6966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b578a043b76009676e6a33db020273a817640bf5891aec6f8f2b46dd0916508b" Oct 30 18:08:55.022019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.021823 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e1df189bb31fb8c419c1e848482afda4dccb516290aeca4a9ece71e97496zph" Oct 30 18:08:55.023741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.023711 6966 generic.go:356] "Generic (PLEG): container finished" podID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerID="10a4360c62356cff379ccf6e94a9f9bd345bd87136f192b733b45e4d8d3387cb" exitCode=0 Oct 30 18:08:55.023859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.023797 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" event={"ID":"fb92d224-a3ba-4114-87e2-bf7135008b3a","Type":"ContainerDied","Data":"10a4360c62356cff379ccf6e94a9f9bd345bd87136f192b733b45e4d8d3387cb"} Oct 30 18:08:55.026227 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.026198 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" event={"ID":"a4be9e88-0385-4c8b-a2fb-c48fd1403898","Type":"ContainerDied","Data":"a769d799b645108510c271712f53b6c4b13917d605dd5b6e7ecae341cafeeb9d"} Oct 30 18:08:55.026335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.026252 6966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a769d799b645108510c271712f53b6c4b13917d605dd5b6e7ecae341cafeeb9d" Oct 30 18:08:55.026335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:55.026211 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6fb1a13d619f6ee73e221b33f4898cc831d48b9998b06378838f67eccfr4ldm" Oct 30 18:08:56.032310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:56.032279 6966 generic.go:356] "Generic (PLEG): container finished" podID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerID="540daef40ad3c3cfe9bf223853f3fdb963e0575402de10d8a3675bd4361a03ba" exitCode=0 Oct 30 18:08:56.032720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:56.032359 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" event={"ID":"fb92d224-a3ba-4114-87e2-bf7135008b3a","Type":"ContainerDied","Data":"540daef40ad3c3cfe9bf223853f3fdb963e0575402de10d8a3675bd4361a03ba"} Oct 30 18:08:56.672665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:56.672634 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:56.887742 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:56.887701 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:57.152530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.152509 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:57.201862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.201837 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util\") pod \"fb92d224-a3ba-4114-87e2-bf7135008b3a\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " Oct 30 18:08:57.202005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.201880 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle\") pod \"fb92d224-a3ba-4114-87e2-bf7135008b3a\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " Oct 30 18:08:57.202005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.201929 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc\" (UniqueName: \"kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc\") pod \"fb92d224-a3ba-4114-87e2-bf7135008b3a\" (UID: \"fb92d224-a3ba-4114-87e2-bf7135008b3a\") " Oct 30 18:08:57.202405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.202380 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle" pod "fb92d224-a3ba-4114-87e2-bf7135008b3a" (UID: "fb92d224-a3ba-4114-87e2-bf7135008b3a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:57.203925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.203903 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc" pod "fb92d224-a3ba-4114-87e2-bf7135008b3a" (UID: "fb92d224-a3ba-4114-87e2-bf7135008b3a"). InnerVolumeSpecName "kube-api-access-9vfqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:08:57.207010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.206987 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util" pod "fb92d224-a3ba-4114-87e2-bf7135008b3a" (UID: "fb92d224-a3ba-4114-87e2-bf7135008b3a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:08:57.302874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.302812 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vfqc\" (UniqueName: \"kubernetes.io/projected/fb92d224-a3ba-4114-87e2-bf7135008b3a-kube-api-access-9vfqc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:57.302874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.302846 6966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-util\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:57.302874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.302856 6966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fb92d224-a3ba-4114-87e2-bf7135008b3a-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:08:57.745525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.745486 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:57.987455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:57.987419 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:58.041643 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:58.041578 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" Oct 30 18:08:58.041643 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:58.041587 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/07f0b9f07fd02dc1509b226b57404b9c198c3c836737a4f34809a100544sfnz" event={"ID":"fb92d224-a3ba-4114-87e2-bf7135008b3a","Type":"ContainerDied","Data":"1561752cb7094ce765e123bdc62491348977ec3a7054b2d562d91e2931b32b12"} Oct 30 18:08:58.041643 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:58.041626 6966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1561752cb7094ce765e123bdc62491348977ec3a7054b2d562d91e2931b32b12" Oct 30 18:08:58.744923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:58.744887 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:58.991023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:58.990986 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:08:59.221053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:08:59.221025 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:01.151784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.151751 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:01.188907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.188875 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs"] Oct 30 18:09:01.189238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189222 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="extract" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189240 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="extract" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189265 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="extract" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189275 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="extract" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189287 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="pull" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189295 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="pull" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189309 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="util" Oct 30 18:09:01.189331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189316 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="util" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189339 6966 cpu_manager.go:414] "containerMap: removing container" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="extract" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189346 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="extract" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189357 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="util" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189366 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="util" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189378 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="pull" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189385 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="pull" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189402 6966 cpu_manager.go:414] "containerMap: removing container" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="util" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189409 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="util" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189427 6966 cpu_manager.go:414] "containerMap: removing container" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="pull" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189434 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="pull" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189531 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="513bcca5-acad-46e5-ab3e-9752c22cc0da" containerName="extract" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189549 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4be9e88-0385-4c8b-a2fb-c48fd1403898" containerName="extract" Oct 30 18:09:01.189735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.189561 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb92d224-a3ba-4114-87e2-bf7135008b3a" containerName="extract" Oct 30 18:09:01.192826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.192807 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.207137 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.207093 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:09:01.218279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.218260 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-controller-manager-dockercfg-85bvp\"" Oct 30 18:09:01.218361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.218279 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-metrics\"" Oct 30 18:09:01.218361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.218332 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-opentelemetry-operator\"/\"kube-root-ca.crt\"" Oct 30 18:09:01.219754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.219737 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs"] Oct 30 18:09:01.233330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.233304 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d0ca190-29b2-445e-9d4b-9478d317e885-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs\" (UID: \"5d0ca190-29b2-445e-9d4b-9478d317e885\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.233477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.233370 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrcp6\" (UniqueName: \"kubernetes.io/projected/5d0ca190-29b2-445e-9d4b-9478d317e885-kube-api-access-qrcp6\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs\" (UID: \"5d0ca190-29b2-445e-9d4b-9478d317e885\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.334823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.334789 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d0ca190-29b2-445e-9d4b-9478d317e885-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs\" (UID: \"5d0ca190-29b2-445e-9d4b-9478d317e885\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.334931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.334842 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrcp6\" (UniqueName: \"kubernetes.io/projected/5d0ca190-29b2-445e-9d4b-9478d317e885-kube-api-access-qrcp6\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs\" (UID: \"5d0ca190-29b2-445e-9d4b-9478d317e885\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.337110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.337084 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5d0ca190-29b2-445e-9d4b-9478d317e885-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs\" (UID: \"5d0ca190-29b2-445e-9d4b-9478d317e885\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.357260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.357238 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrcp6\" (UniqueName: \"kubernetes.io/projected/5d0ca190-29b2-445e-9d4b-9478d317e885-kube-api-access-qrcp6\") pod \"opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs\" (UID: \"5d0ca190-29b2-445e-9d4b-9478d317e885\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.504032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.503976 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:01.765833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.765768 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq"] Oct 30 18:09:01.769494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.769452 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.798173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.798155 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-opentelemetry-operator\"/\"opentelemetry-operator-webhook-service-cert\"" Oct 30 18:09:01.799507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.799488 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq"] Oct 30 18:09:01.837716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.837687 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs"] Oct 30 18:09:01.840015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.839992 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e837be1-9009-4726-beaa-bcd07a2a0e44-apiservice-cert\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.840120 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.840048 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e837be1-9009-4726-beaa-bcd07a2a0e44-webhook-cert\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.840120 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.840098 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kgx2\" (UniqueName: \"kubernetes.io/projected/5e837be1-9009-4726-beaa-bcd07a2a0e44-kube-api-access-5kgx2\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.841919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.841895 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:01.843869 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:09:01.843834 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d0ca190_29b2_445e_9d4b_9478d317e885.slice/crio-6c63de688f4a1e5d52d18ee13c7391799dd78ffede42514c20fd38d70fea9470 WatchSource:0}: Error finding container 6c63de688f4a1e5d52d18ee13c7391799dd78ffede42514c20fd38d70fea9470: Status 404 returned error can't find the container with id 6c63de688f4a1e5d52d18ee13c7391799dd78ffede42514c20fd38d70fea9470 Oct 30 18:09:01.941399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.941374 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e837be1-9009-4726-beaa-bcd07a2a0e44-webhook-cert\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.941542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.941405 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kgx2\" (UniqueName: \"kubernetes.io/projected/5e837be1-9009-4726-beaa-bcd07a2a0e44-kube-api-access-5kgx2\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.941542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.941506 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e837be1-9009-4726-beaa-bcd07a2a0e44-apiservice-cert\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.943736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.943715 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5e837be1-9009-4726-beaa-bcd07a2a0e44-webhook-cert\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.943822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.943740 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5e837be1-9009-4726-beaa-bcd07a2a0e44-apiservice-cert\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:01.963392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:01.963371 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kgx2\" (UniqueName: \"kubernetes.io/projected/5e837be1-9009-4726-beaa-bcd07a2a0e44-kube-api-access-5kgx2\") pod \"opentelemetry-operator-webhook-79fb9f5d66-8x9cq\" (UID: \"5e837be1-9009-4726-beaa-bcd07a2a0e44\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:02.060395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:02.060325 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" event={"ID":"5d0ca190-29b2-445e-9d4b-9478d317e885","Type":"ContainerStarted","Data":"6c63de688f4a1e5d52d18ee13c7391799dd78ffede42514c20fd38d70fea9470"} Oct 30 18:09:02.082499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:02.082476 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:02.104296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:02.104269 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:02.396910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:02.396882 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq"] Oct 30 18:09:02.401775 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:09:02.401752 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e837be1_9009_4726_beaa_bcd07a2a0e44.slice/crio-20d1edbbbb6eb8a0ab50731352f9079bd74f027f5d4bb08c67a86cac2730c5b7 WatchSource:0}: Error finding container 20d1edbbbb6eb8a0ab50731352f9079bd74f027f5d4bb08c67a86cac2730c5b7: Status 404 returned error can't find the container with id 20d1edbbbb6eb8a0ab50731352f9079bd74f027f5d4bb08c67a86cac2730c5b7 Oct 30 18:09:02.887277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:02.887244 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:03.065640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:03.065608 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" event={"ID":"5e837be1-9009-4726-beaa-bcd07a2a0e44","Type":"ContainerStarted","Data":"20d1edbbbb6eb8a0ab50731352f9079bd74f027f5d4bb08c67a86cac2730c5b7"} Oct 30 18:09:03.151360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:03.151264 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:03.886676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:03.886642 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:04.103007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:04.102974 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:05.077917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.077868 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" event={"ID":"5e837be1-9009-4726-beaa-bcd07a2a0e44","Type":"ContainerStarted","Data":"7144231ba4f0f60fb772f3718ce8507bf80d4f7b454b82199b0476a8b0fc175d"} Oct 30 18:09:05.078354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.078034 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:05.079593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.079569 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" event={"ID":"5d0ca190-29b2-445e-9d4b-9478d317e885","Type":"ContainerStarted","Data":"a0419fdd236133b0b718b6b9b4acb6e803c5b8a793c78bfdaad090e801370535"} Oct 30 18:09:05.079712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.079621 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:05.143830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.143778 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" podStartSLOduration=2.179997624 podStartE2EDuration="4.143761779s" totalImagesPullingTime="1.963764155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:05.131614969 +0000 UTC m=+707.837800325" watchObservedRunningTime="2026-10-30 18:09:05.143761779 +0000 UTC m=+707.849947138" Oct 30 18:09:05.177263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.177220 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" podStartSLOduration=1.657475629 podStartE2EDuration="4.177207668s" totalImagesPullingTime="2.519732039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:05.175963429 +0000 UTC m=+707.882148787" watchObservedRunningTime="2026-10-30 18:09:05.177207668 +0000 UTC m=+707.883393024" Oct 30 18:09:05.187574 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.187524 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:05.404094 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.404016 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:05.959680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:05.959645 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:06.749255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:06.749220 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:07.150815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:07.150786 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:07.422792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:07.422715 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:07.724564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:07.724482 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:07.958945 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:07.958912 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:08.211140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:08.211107 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:08.552646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:08.552562 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:08.777367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:08.777339 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:09.087068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:09.087042 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:09.793619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:09.793587 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.520043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:11.520007 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:11.762967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:11.762936 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.372775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.372738 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.575475 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.575426 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.775279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.775201 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h"] Oct 30 18:09:12.778708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.778684 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" Oct 30 18:09:12.794075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.794053 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:09:12.794217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.794204 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-observability-operator\"/\"kube-root-ca.crt\"" Oct 30 18:09:12.804302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.804273 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-dockercfg-nvlhz\"" Oct 30 18:09:12.805943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.805916 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h"] Oct 30 18:09:12.807639 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.807614 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:12.829268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.829240 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkmqx\" (UniqueName: \"kubernetes.io/projected/78d4c6fd-b266-407d-8c41-4b5e38abee05-kube-api-access-jkmqx\") pod \"obo-prometheus-operator-6ddb664c4f-qq76h\" (UID: \"78d4c6fd-b266-407d-8c41-4b5e38abee05\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" Oct 30 18:09:12.929892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.929863 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkmqx\" (UniqueName: \"kubernetes.io/projected/78d4c6fd-b266-407d-8c41-4b5e38abee05-kube-api-access-jkmqx\") pod \"obo-prometheus-operator-6ddb664c4f-qq76h\" (UID: \"78d4c6fd-b266-407d-8c41-4b5e38abee05\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" Oct 30 18:09:12.952029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:12.952005 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkmqx\" (UniqueName: \"kubernetes.io/projected/78d4c6fd-b266-407d-8c41-4b5e38abee05-kube-api-access-jkmqx\") pod \"obo-prometheus-operator-6ddb664c4f-qq76h\" (UID: \"78d4c6fd-b266-407d-8c41-4b5e38abee05\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" Oct 30 18:09:13.022628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.022590 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:13.090923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.090894 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" Oct 30 18:09:13.241016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.240991 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7"] Oct 30 18:09:13.245624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.245604 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.267119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.267096 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7"] Oct 30 18:09:13.313822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.313794 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:13.330291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.330273 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-service-cert\"" Oct 30 18:09:13.330567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.330548 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-observability-operator\"/\"obo-prometheus-operator-admission-webhook-dockercfg-tdgcq\"" Oct 30 18:09:13.333874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.333856 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7\" (UID: \"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.333941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.333899 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7\" (UID: \"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.434866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.434811 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h"] Oct 30 18:09:13.435028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.434960 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7\" (UID: \"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.435103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.435043 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7\" (UID: \"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.437675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.437633 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7-webhook-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7\" (UID: \"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.437727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.437691 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7-apiservice-cert\") pod \"obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7\" (UID: \"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7\") " pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.439943 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:09:13.439918 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d4c6fd_b266_407d_8c41_4b5e38abee05.slice/crio-d383f5f6b459c47c0c64d72f56af2fb5a0c228e9fcdf5c2690670e465a71ba49 WatchSource:0}: Error finding container d383f5f6b459c47c0c64d72f56af2fb5a0c228e9fcdf5c2690670e465a71ba49: Status 404 returned error can't find the container with id d383f5f6b459c47c0c64d72f56af2fb5a0c228e9fcdf5c2690670e465a71ba49 Oct 30 18:09:13.557991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.557967 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" Oct 30 18:09:13.868686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.868653 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7"] Oct 30 18:09:13.873514 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:09:13.873480 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a37f580_6998_4dd2_9cc5_bfe77ee5a2f7.slice/crio-376d9e648a97275382bf2bd30386b41212b74be6a9bfa4ce0ac748275bde279f WatchSource:0}: Error finding container 376d9e648a97275382bf2bd30386b41212b74be6a9bfa4ce0ac748275bde279f: Status 404 returned error can't find the container with id 376d9e648a97275382bf2bd30386b41212b74be6a9bfa4ce0ac748275bde279f Oct 30 18:09:13.940755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:13.940729 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:14.121085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:14.121002 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" event={"ID":"78d4c6fd-b266-407d-8c41-4b5e38abee05","Type":"ContainerStarted","Data":"d383f5f6b459c47c0c64d72f56af2fb5a0c228e9fcdf5c2690670e465a71ba49"} Oct 30 18:09:14.122351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:14.122331 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" event={"ID":"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7","Type":"ContainerStarted","Data":"376d9e648a97275382bf2bd30386b41212b74be6a9bfa4ce0ac748275bde279f"} Oct 30 18:09:16.086296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:16.086260 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-5f695b7fbf-ldxhs" Oct 30 18:09:16.088086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:16.088062 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-webhook-79fb9f5d66-8x9cq" Oct 30 18:09:16.497566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:16.497487 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:16.781400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:16.781252 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.005998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:17.005964 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:17.140083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:17.140057 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" event={"ID":"78d4c6fd-b266-407d-8c41-4b5e38abee05","Type":"ContainerStarted","Data":"5db3a920e8e996a392104fa2566ebad58fa88cb41a46668b93af69889e4adfc4"} Oct 30 18:09:17.141752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:17.141729 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" event={"ID":"1a37f580-6998-4dd2-9cc5-bfe77ee5a2f7","Type":"ContainerStarted","Data":"f5e93caab4bc8412362264a78b3c5ed365a33583231d2308da3744d9acc73663"} Oct 30 18:09:17.205404 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:09:17.205369 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:09:17.234530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:17.234483 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-6ddb664c4f-qq76h" podStartSLOduration=2.044053469 podStartE2EDuration="5.234470287s" totalImagesPullingTime="3.190416818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:17.232970507 +0000 UTC m=+719.939155863" watchObservedRunningTime="2026-10-30 18:09:17.234470287 +0000 UTC m=+719.940655639" Oct 30 18:09:17.295921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:17.295849 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-observability-operator/obo-prometheus-operator-admission-webhook-f4675f76b-lrrk7" podStartSLOduration=1.188697425 podStartE2EDuration="4.295836912s" totalImagesPullingTime="3.107139487s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:17.294685899 +0000 UTC m=+720.000871256" watchObservedRunningTime="2026-10-30 18:09:17.295836912 +0000 UTC m=+720.002022270" Oct 30 18:09:17.952733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:17.952693 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:20.973604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:20.973563 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.174631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:21.174594 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.390102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:21.390070 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:21.587936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:21.587898 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:22.167215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:22.167181 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:22.399330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:22.399294 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:22.607965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:22.607931 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:22.819071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:22.819035 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:25.194149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:25.194115 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:35.173544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.173449 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g"] Oct 30 18:09:35.176970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.176949 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.210415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.210394 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g"] Oct 30 18:09:35.319717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.319692 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7t9p\" (UniqueName: \"kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p\") pod \"opentelemetry-operator-controller-manager-84466c79dd-mm29g\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.319851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.319739 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-84466c79dd-mm29g\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.420379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.420352 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7t9p\" (UniqueName: \"kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p\") pod \"opentelemetry-operator-controller-manager-84466c79dd-mm29g\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.420523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.420397 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-84466c79dd-mm29g\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.422674 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.422653 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert\") pod \"opentelemetry-operator-controller-manager-84466c79dd-mm29g\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.442899 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.442846 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7t9p\" (UniqueName: \"kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p\") pod \"opentelemetry-operator-controller-manager-84466c79dd-mm29g\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.488380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.488360 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:35.734121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.734047 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:35.813429 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.813391 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g"] Oct 30 18:09:35.819563 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:09:35.819537 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf46c0617_be9e_4380_ae91_7ff01c96cee3.slice/crio-438a6169c9a5c2edf84a4ea037a8297e6a4281e1f21defc5dcfbd6173e8a51d2 WatchSource:0}: Error finding container 438a6169c9a5c2edf84a4ea037a8297e6a4281e1f21defc5dcfbd6173e8a51d2: Status 404 returned error can't find the container with id 438a6169c9a5c2edf84a4ea037a8297e6a4281e1f21defc5dcfbd6173e8a51d2 Oct 30 18:09:35.971264 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:35.971224 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:36.231694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.231662 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" event={"ID":"f46c0617-be9e-4380-ae91-7ff01c96cee3","Type":"ContainerStarted","Data":"aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281"} Oct 30 18:09:36.232096 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.231708 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" event={"ID":"f46c0617-be9e-4380-ae91-7ff01c96cee3","Type":"ContainerStarted","Data":"438a6169c9a5c2edf84a4ea037a8297e6a4281e1f21defc5dcfbd6173e8a51d2"} Oct 30 18:09:36.232096 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.231778 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:36.285837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.285794 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" podStartSLOduration=1.285782412 podStartE2EDuration="1.285782412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:36.285135928 +0000 UTC m=+738.991321285" watchObservedRunningTime="2026-10-30 18:09:36.285782412 +0000 UTC m=+738.991967770" Oct 30 18:09:36.698281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.698246 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g"] Oct 30 18:09:36.839883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.839848 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr"] Oct 30 18:09:36.842349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.842328 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:36.856097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.856078 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"d28b81ce69fab5e58d2cd73462dde4041bbc20-487d35f4-dockercfg-xjnqd\"" Oct 30 18:09:36.856371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.856357 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:09:36.856416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.856378 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:09:36.865173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.865153 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr"] Oct 30 18:09:36.932782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.932750 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:36.932985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.932789 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:36.932985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:36.932949 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jv2s\" (UniqueName: \"kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.033743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.033658 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jv2s\" (UniqueName: \"kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.033743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.033715 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.033936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.033747 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.034148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.034130 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.034200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.034177 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.061513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.061488 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jv2s\" (UniqueName: \"kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s\") pod \"d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.154939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.154908 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:37.451095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:37.451056 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr"] Oct 30 18:09:38.243343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.243311 6966 generic.go:356] "Generic (PLEG): container finished" podID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerID="ebc97f0a3451b1d1de214be583477024aa9821121d36d98cb1c421a858fe4e9d" exitCode=0 Oct 30 18:09:38.243520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.243402 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" event={"ID":"469b6899-9b90-48c9-9ca5-bc6e4d261dfb","Type":"ContainerDied","Data":"ebc97f0a3451b1d1de214be583477024aa9821121d36d98cb1c421a858fe4e9d"} Oct 30 18:09:38.243520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.243475 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" event={"ID":"469b6899-9b90-48c9-9ca5-bc6e4d261dfb","Type":"ContainerStarted","Data":"f7c272f77a8f680d22c018c7292fe5abebd7d269cc7b75488ba04efe23cc599d"} Oct 30 18:09:38.243605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.243525 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" podUID="f46c0617-be9e-4380-ae91-7ff01c96cee3" containerName="manager" containerID="cri-o://aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281" gracePeriod=10 Oct 30 18:09:38.276375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.276351 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:38.476360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.476340 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:38.502673 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.502589 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:38.645976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.645951 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p\" (UniqueName: \"kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p\") pod \"f46c0617-be9e-4380-ae91-7ff01c96cee3\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " Oct 30 18:09:38.646112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.645991 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert\") pod \"f46c0617-be9e-4380-ae91-7ff01c96cee3\" (UID: \"f46c0617-be9e-4380-ae91-7ff01c96cee3\") " Oct 30 18:09:38.648132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.648095 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert" pod "f46c0617-be9e-4380-ae91-7ff01c96cee3" (UID: "f46c0617-be9e-4380-ae91-7ff01c96cee3"). InnerVolumeSpecName "opentelemetry-operator-metrics-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:38.648226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.648140 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p" pod "f46c0617-be9e-4380-ae91-7ff01c96cee3" (UID: "f46c0617-be9e-4380-ae91-7ff01c96cee3"). InnerVolumeSpecName "kube-api-access-h7t9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:38.747400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.747367 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7t9p\" (UniqueName: \"kubernetes.io/projected/f46c0617-be9e-4380-ae91-7ff01c96cee3-kube-api-access-h7t9p\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:09:38.747400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.747396 6966 reconciler_common.go:299] "Volume detached for volume \"opentelemetry-operator-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f46c0617-be9e-4380-ae91-7ff01c96cee3-opentelemetry-operator-metrics-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:09:38.917078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:38.917042 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:39.140251 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.140217 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:39.249774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.249705 6966 generic.go:356] "Generic (PLEG): container finished" podID="f46c0617-be9e-4380-ae91-7ff01c96cee3" containerID="aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281" exitCode=0 Oct 30 18:09:39.249774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.249748 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" Oct 30 18:09:39.249956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.249743 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" event={"ID":"f46c0617-be9e-4380-ae91-7ff01c96cee3","Type":"ContainerDied","Data":"aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281"} Oct 30 18:09:39.249956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.249862 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g" event={"ID":"f46c0617-be9e-4380-ae91-7ff01c96cee3","Type":"ContainerDied","Data":"438a6169c9a5c2edf84a4ea037a8297e6a4281e1f21defc5dcfbd6173e8a51d2"} Oct 30 18:09:39.249956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.249884 6966 scope.go:122] "RemoveContainer" containerID="aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281" Oct 30 18:09:39.259174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.258986 6966 scope.go:122] "RemoveContainer" containerID="aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281" Oct 30 18:09:39.259285 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:09:39.259261 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281\": container with ID starting with aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281 not found: ID does not exist" containerID="aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281" Oct 30 18:09:39.259351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.259297 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281"} err="failed to get container status \"aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281\": rpc error: code = NotFound desc = could not find container \"aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281\": container with ID starting with aed40e969408c3bba97533cc50dd87d28c9bc573828dcdef4a3b4ba81aec9281 not found: ID does not exist" Oct 30 18:09:39.313345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.313316 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g"] Oct 30 18:09:39.324346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.324321 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-opentelemetry-operator/opentelemetry-operator-controller-manager-84466c79dd-mm29g"] Oct 30 18:09:39.411399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:39.411375 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f46c0617-be9e-4380-ae91-7ff01c96cee3" path="/var/lib/kubelet/pods/f46c0617-be9e-4380-ae91-7ff01c96cee3/volumes" Oct 30 18:09:40.256752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:40.256718 6966 generic.go:356] "Generic (PLEG): container finished" podID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerID="38576141afe4fdddfa720b3546b09f7934f1e5a5544234631c664383243133bc" exitCode=0 Oct 30 18:09:40.257207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:40.256812 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" event={"ID":"469b6899-9b90-48c9-9ca5-bc6e4d261dfb","Type":"ContainerDied","Data":"38576141afe4fdddfa720b3546b09f7934f1e5a5544234631c664383243133bc"} Oct 30 18:09:40.264819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:40.264790 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:40.547493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:40.547412 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:41.070968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:41.070934 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:41.265927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:41.265887 6966 generic.go:356] "Generic (PLEG): container finished" podID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerID="1b290cebff7f40a0c955ce0dbff43012512fa9bf92812d4fb0f1cd1bb4736ced" exitCode=0 Oct 30 18:09:41.266362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:41.265989 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" event={"ID":"469b6899-9b90-48c9-9ca5-bc6e4d261dfb","Type":"ContainerDied","Data":"1b290cebff7f40a0c955ce0dbff43012512fa9bf92812d4fb0f1cd1bb4736ced"} Oct 30 18:09:41.293843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:41.293812 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:42.388655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.388633 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:42.478649 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.478613 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s\" (UniqueName: \"kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s\") pod \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " Oct 30 18:09:42.478827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.478710 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle\") pod \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " Oct 30 18:09:42.478827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.478783 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util\") pod \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\" (UID: \"469b6899-9b90-48c9-9ca5-bc6e4d261dfb\") " Oct 30 18:09:42.480288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.480234 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle" pod "469b6899-9b90-48c9-9ca5-bc6e4d261dfb" (UID: "469b6899-9b90-48c9-9ca5-bc6e4d261dfb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:42.483507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.481232 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s" pod "469b6899-9b90-48c9-9ca5-bc6e4d261dfb" (UID: "469b6899-9b90-48c9-9ca5-bc6e4d261dfb"). InnerVolumeSpecName "kube-api-access-6jv2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:42.486339 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.486309 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util" pod "469b6899-9b90-48c9-9ca5-bc6e4d261dfb" (UID: "469b6899-9b90-48c9-9ca5-bc6e4d261dfb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:09:42.580085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.580053 6966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-util\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:09:42.580085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.580080 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jv2s\" (UniqueName: \"kubernetes.io/projected/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-kube-api-access-6jv2s\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:09:42.580085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:42.580091 6966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/469b6899-9b90-48c9-9ca5-bc6e4d261dfb-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:09:43.277078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:43.277046 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" event={"ID":"469b6899-9b90-48c9-9ca5-bc6e4d261dfb","Type":"ContainerDied","Data":"f7c272f77a8f680d22c018c7292fe5abebd7d269cc7b75488ba04efe23cc599d"} Oct 30 18:09:43.277078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:43.277063 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/d28b81ce69fab5e58d2cd73462dde4041bbc20368f72658a361322aaadzw7gr" Oct 30 18:09:43.277277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:43.277087 6966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7c272f77a8f680d22c018c7292fe5abebd7d269cc7b75488ba04efe23cc599d" Oct 30 18:09:47.198727 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:09:47.198690 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:09:48.596058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:48.595927 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:48.796749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:48.796714 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:49.269806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:49.269771 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:49.528896 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:49.528805 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:52.663261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:52.663223 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.163646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:53.163611 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:53.373873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:53.373843 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:54.863553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:54.863526 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:55.068977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:55.068935 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:55.270547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:55.270449 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:55.494957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:55.494920 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:56.868776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:56.868740 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:09:57.069792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:09:57.069760 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:00.059276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:00.059239 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:00.253498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:00.253447 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:04.266411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:04.266375 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:04.527586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:04.527502 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.674638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:07.674603 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:07.890362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:07.890324 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:17.198793 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:10:17.198758 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:10:18.002294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:18.002259 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:20.792266 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:20.792229 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:20.998748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:20.998714 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:21.616863 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.616829 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x"] Oct 30 18:10:21.617150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617139 6966 cpu_manager.go:414] "containerMap: removing container" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="util" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617152 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="util" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617172 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f46c0617-be9e-4380-ae91-7ff01c96cee3" containerName="manager" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617177 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f46c0617-be9e-4380-ae91-7ff01c96cee3" containerName="manager" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617192 6966 cpu_manager.go:414] "containerMap: removing container" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="pull" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617198 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="pull" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617204 6966 cpu_manager.go:414] "containerMap: removing container" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="extract" Oct 30 18:10:21.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617209 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="extract" Oct 30 18:10:21.617418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617261 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f46c0617-be9e-4380-ae91-7ff01c96cee3" containerName="manager" Oct 30 18:10:21.617418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.617269 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="469b6899-9b90-48c9-9ca5-bc6e4d261dfb" containerName="extract" Oct 30 18:10:21.622104 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.622083 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.647226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.647201 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:10:21.647330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.647237 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:10:21.647394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.647380 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"b60fd1021875ce556623478c7c5d4d22148dff-4f0c4bdd-dockercfg-ssvtl\"" Oct 30 18:10:21.648751 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.648731 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x"] Oct 30 18:10:21.670018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.669991 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.670113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.670049 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.670157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.670130 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42kpn\" (UniqueName: \"kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.770897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.770876 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42kpn\" (UniqueName: \"kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.770997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.770926 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.770997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.770959 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.771237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.771217 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.771293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.771251 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.793883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.793862 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42kpn\" (UniqueName: \"kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn\") pod \"b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:21.937732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:21.937685 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:22.230769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:22.230702 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x"] Oct 30 18:10:22.236067 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:10:22.236033 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb13b004f_e652_4e49_8222_a3e6edf5bcbd.slice/crio-fa9cd0effd117ed0e8bda04c23f9c2bdea44e919b796345ed04cd7d0cc8bd5aa WatchSource:0}: Error finding container fa9cd0effd117ed0e8bda04c23f9c2bdea44e919b796345ed04cd7d0cc8bd5aa: Status 404 returned error can't find the container with id fa9cd0effd117ed0e8bda04c23f9c2bdea44e919b796345ed04cd7d0cc8bd5aa Oct 30 18:10:22.452855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:22.452824 6966 generic.go:356] "Generic (PLEG): container finished" podID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerID="de98ff6ca7ba664f3b41911e8ef7b75f2376974f593c251c18030060fd8d0477" exitCode=0 Oct 30 18:10:22.452983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:22.452908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" event={"ID":"b13b004f-e652-4e49-8222-a3e6edf5bcbd","Type":"ContainerDied","Data":"de98ff6ca7ba664f3b41911e8ef7b75f2376974f593c251c18030060fd8d0477"} Oct 30 18:10:22.452983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:22.452962 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" event={"ID":"b13b004f-e652-4e49-8222-a3e6edf5bcbd","Type":"ContainerStarted","Data":"fa9cd0effd117ed0e8bda04c23f9c2bdea44e919b796345ed04cd7d0cc8bd5aa"} Oct 30 18:10:24.464074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:24.464038 6966 generic.go:356] "Generic (PLEG): container finished" podID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerID="665f5c5673de1fcfbe3044d73175be2f5605c56621b86f02bfcb5ea353da955f" exitCode=0 Oct 30 18:10:24.464630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:24.464084 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" event={"ID":"b13b004f-e652-4e49-8222-a3e6edf5bcbd","Type":"ContainerDied","Data":"665f5c5673de1fcfbe3044d73175be2f5605c56621b86f02bfcb5ea353da955f"} Oct 30 18:10:25.470087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:25.470054 6966 generic.go:356] "Generic (PLEG): container finished" podID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerID="b1c32636f0078b8dd05fea24f5b6748793bb0a025dcb8c070cc3b0748b0ff063" exitCode=0 Oct 30 18:10:25.470494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:25.470131 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" event={"ID":"b13b004f-e652-4e49-8222-a3e6edf5bcbd","Type":"ContainerDied","Data":"b1c32636f0078b8dd05fea24f5b6748793bb0a025dcb8c070cc3b0748b0ff063"} Oct 30 18:10:26.593127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.593104 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:26.708181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.708150 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util\") pod \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " Oct 30 18:10:26.708341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.708214 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle\") pod \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " Oct 30 18:10:26.708341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.708277 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn\" (UniqueName: \"kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn\") pod \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\" (UID: \"b13b004f-e652-4e49-8222-a3e6edf5bcbd\") " Oct 30 18:10:26.708802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.708773 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle" pod "b13b004f-e652-4e49-8222-a3e6edf5bcbd" (UID: "b13b004f-e652-4e49-8222-a3e6edf5bcbd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:26.710328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.710300 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn" pod "b13b004f-e652-4e49-8222-a3e6edf5bcbd" (UID: "b13b004f-e652-4e49-8222-a3e6edf5bcbd"). InnerVolumeSpecName "kube-api-access-42kpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:10:26.714229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.714182 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util" pod "b13b004f-e652-4e49-8222-a3e6edf5bcbd" (UID: "b13b004f-e652-4e49-8222-a3e6edf5bcbd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:10:26.808800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.808734 6966 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:10:26.808800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.808761 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42kpn\" (UniqueName: \"kubernetes.io/projected/b13b004f-e652-4e49-8222-a3e6edf5bcbd-kube-api-access-42kpn\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:10:26.808800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:26.808772 6966 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b13b004f-e652-4e49-8222-a3e6edf5bcbd-util\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:10:27.484224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:27.484189 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" Oct 30 18:10:27.484374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:27.484227 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/b60fd1021875ce556623478c7c5d4d22148dffc3b09ba9d735533c6722l5q8x" event={"ID":"b13b004f-e652-4e49-8222-a3e6edf5bcbd","Type":"ContainerDied","Data":"fa9cd0effd117ed0e8bda04c23f9c2bdea44e919b796345ed04cd7d0cc8bd5aa"} Oct 30 18:10:27.484374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:27.484298 6966 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa9cd0effd117ed0e8bda04c23f9c2bdea44e919b796345ed04cd7d0cc8bd5aa" Oct 30 18:10:32.218558 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:32.218519 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:32.431296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:32.431262 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.457789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:33.457757 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:33.672602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:33.672568 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:36.575511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.573432 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:36.721148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721113 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-mb45t"] Oct 30 18:10:36.721484 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721456 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="util" Oct 30 18:10:36.721543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721485 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="util" Oct 30 18:10:36.721543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721496 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="extract" Oct 30 18:10:36.721543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721502 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="extract" Oct 30 18:10:36.721543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721515 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="pull" Oct 30 18:10:36.721543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721527 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="pull" Oct 30 18:10:36.721695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.721615 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b13b004f-e652-4e49-8222-a3e6edf5bcbd" containerName="extract" Oct 30 18:10:36.724804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.724783 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:36.741946 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.741924 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"kube-root-ca.crt\"" Oct 30 18:10:36.742143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.741961 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"openshift-service-ca.crt\"" Oct 30 18:10:36.753157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.753134 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"cluster-logging-operator-metrics-certs\"" Oct 30 18:10:36.753157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.753139 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"cluster-logging-operator-dockercfg-zbs9r\"" Oct 30 18:10:36.758475 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.758446 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-mb45t"] Oct 30 18:10:36.798382 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.798357 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fnh2\" (UniqueName: \"kubernetes.io/projected/7ed61682-87dd-4914-a213-a613c031391d-kube-api-access-6fnh2\") pod \"cluster-logging-operator-5d6db4c85-mb45t\" (UID: \"7ed61682-87dd-4914-a213-a613c031391d\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:36.798512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.798413 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed61682-87dd-4914-a213-a613c031391d-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-mb45t\" (UID: \"7ed61682-87dd-4914-a213-a613c031391d\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:36.899693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.899629 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed61682-87dd-4914-a213-a613c031391d-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-mb45t\" (UID: \"7ed61682-87dd-4914-a213-a613c031391d\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:36.899812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.899715 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fnh2\" (UniqueName: \"kubernetes.io/projected/7ed61682-87dd-4914-a213-a613c031391d-kube-api-access-6fnh2\") pod \"cluster-logging-operator-5d6db4c85-mb45t\" (UID: \"7ed61682-87dd-4914-a213-a613c031391d\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:36.902001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.901973 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ed61682-87dd-4914-a213-a613c031391d-metrics-certs\") pod \"cluster-logging-operator-5d6db4c85-mb45t\" (UID: \"7ed61682-87dd-4914-a213-a613c031391d\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:36.922341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:36.922314 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fnh2\" (UniqueName: \"kubernetes.io/projected/7ed61682-87dd-4914-a213-a613c031391d-kube-api-access-6fnh2\") pod \"cluster-logging-operator-5d6db4c85-mb45t\" (UID: \"7ed61682-87dd-4914-a213-a613c031391d\") " pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:37.036367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:37.036346 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" Oct 30 18:10:37.349158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:37.349132 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/cluster-logging-operator-5d6db4c85-mb45t"] Oct 30 18:10:37.354330 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:10:37.354303 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ed61682_87dd_4914_a213_a613c031391d.slice/crio-ce25dfe4287c662d2c8f6c34c651317f9067a457672e4e41be2edc254a67800c WatchSource:0}: Error finding container ce25dfe4287c662d2c8f6c34c651317f9067a457672e4e41be2edc254a67800c: Status 404 returned error can't find the container with id ce25dfe4287c662d2c8f6c34c651317f9067a457672e4e41be2edc254a67800c Oct 30 18:10:37.535816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:37.535789 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" event={"ID":"7ed61682-87dd-4914-a213-a613c031391d","Type":"ContainerStarted","Data":"ce25dfe4287c662d2c8f6c34c651317f9067a457672e4e41be2edc254a67800c"} Oct 30 18:10:39.258321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:39.258272 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:39.453817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:39.453779 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:41.567167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:41.567136 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" event={"ID":"7ed61682-87dd-4914-a213-a613c031391d","Type":"ContainerStarted","Data":"a071c0a3defbecb3c82efc05ea83926831ef4078cf04be1400eef376399e318c"} Oct 30 18:10:41.610828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:41.610784 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/cluster-logging-operator-5d6db4c85-mb45t" podStartSLOduration=2.400072915 podStartE2EDuration="5.610773939s" totalImagesPullingTime="3.210701024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:41.6095345 +0000 UTC m=+804.315719848" watchObservedRunningTime="2026-10-30 18:10:41.610773939 +0000 UTC m=+804.316959296" Oct 30 18:10:47.199642 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:10:47.199610 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:10:48.659185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:48.659151 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:48.855095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:48.855071 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:53.751164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:53.751128 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:53.966724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:53.966684 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.565828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.565797 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-nlftk"] Oct 30 18:10:58.570581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.570560 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.586714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.586695 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:10:58.586892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.586874 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:10:58.595419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.595398 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.597925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.597908 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:10:58.598066 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.597925 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-dcf4v\"" Oct 30 18:10:58.598145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.597930 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:10:58.599454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.599434 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-nlftk"] Oct 30 18:10:58.665090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.665069 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.665210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.665106 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9mbs\" (UniqueName: \"kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.665210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.665134 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.766532 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.766449 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.766650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.766615 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.766708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.766658 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9mbs\" (UniqueName: \"kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.767273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.767256 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.768840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.768818 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.788937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.788910 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9mbs\" (UniqueName: \"kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs\") pod \"korrel8r-6644769df5-nlftk\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:58.812796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.812773 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:58.883164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:58.883113 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:10:59.177027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.176961 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-nlftk"] Oct 30 18:10:59.181670 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:10:59.181639 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ef22b9_8490_4f5c_8bc6_52b4516cb22e.slice/crio-95f2a840181ab4d86fca28e24cf7f42be1da47e3d42f21d5f65bcfa35ce50990 WatchSource:0}: Error finding container 95f2a840181ab4d86fca28e24cf7f42be1da47e3d42f21d5f65bcfa35ce50990: Status 404 returned error can't find the container with id 95f2a840181ab4d86fca28e24cf7f42be1da47e3d42f21d5f65bcfa35ce50990 Oct 30 18:10:59.479238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.479152 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:10:59.485287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.485255 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.488921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.488887 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:59.503704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.503677 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:10:59.520183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.520160 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:10:59.520183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.520171 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:10:59.520323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.520161 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-l9dsr\"" Oct 30 18:10:59.571683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.571657 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.572024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.571691 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.572024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.571728 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.572024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.571791 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7whf7\" (UniqueName: \"kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.574149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.574130 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:10:59.659690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.659657 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" event={"ID":"17ef22b9-8490-4f5c-8bc6-52b4516cb22e","Type":"ContainerStarted","Data":"95f2a840181ab4d86fca28e24cf7f42be1da47e3d42f21d5f65bcfa35ce50990"} Oct 30 18:10:59.672719 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.672698 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.672840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.672727 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.672840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.672766 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.672840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.672792 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7whf7\" (UniqueName: \"kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.673398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.673377 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.675207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.675182 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.675347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.675332 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.689150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.689129 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:10:59.694454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.694432 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7whf7\" (UniqueName: \"kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7\") pod \"data-science-collector-collector-1\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.802581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.802524 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:10:59.887361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:10:59.887331 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:00.099375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:00.099340 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:00.104516 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:11:00.104482 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d8712a_7364_4250_bc29_c6d0ae3d9938.slice/crio-da451a71e5ad94853b6651669e54baf9817cd996a8c669e4f6eac69935be5916 WatchSource:0}: Error finding container da451a71e5ad94853b6651669e54baf9817cd996a8c669e4f6eac69935be5916: Status 404 returned error can't find the container with id da451a71e5ad94853b6651669e54baf9817cd996a8c669e4f6eac69935be5916 Oct 30 18:11:00.665378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:00.665346 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37d8712a-7364-4250-bc29-c6d0ae3d9938","Type":"ContainerStarted","Data":"da451a71e5ad94853b6651669e54baf9817cd996a8c669e4f6eac69935be5916"} Oct 30 18:11:00.830637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:00.830605 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:01.031991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:01.031945 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:01.812062 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:01.811996 6966 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage2928956623/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/51/5164b26f4cc06f20783d839dc18908e5f81e69881f12b11af759b9da340c3a0a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T181101Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4603776db8046df65b51c9f4298615ba89b42eff09c0828f1a8ac33b26bc6eb6®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=cluster-observability-operator----korrel8r-rhel9&akamai_signature=exp=1793384761~hmac=2e15b4396f04ee861b618660583b8f8a3c414e4fe9b2b1a8d39b675a8ead3301\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" image="registry.redhat.io/cluster-observability-operator/korrel8r-rhel9@sha256:e3ebab37597e15fc58b989e7f3cf70f5561e96723f4f409333e8b99540386c8a" Oct 30 18:11:01.812561 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:01.812271 6966 kuberuntime_manager.go:1707] "Unhandled Error" err="container korrel8r start failed in pod korrel8r-6644769df5-nlftk_redhat-ods-monitoring(17ef22b9-8490-4f5c-8bc6-52b4516cb22e): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \"/var/tmp/container_images_storage2928956623/1\": happened during read: unexpected EOF (while reconnecting: Get \"https://cdn01.quay.io/quayio-production-s3/sha256/51/5164b26f4cc06f20783d839dc18908e5f81e69881f12b11af759b9da340c3a0a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T181101Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4603776db8046df65b51c9f4298615ba89b42eff09c0828f1a8ac33b26bc6eb6®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=cluster-observability-operator----korrel8r-rhel9&akamai_signature=exp=1793384761~hmac=2e15b4396f04ee861b618660583b8f8a3c414e4fe9b2b1a8d39b675a8ead3301\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact" logger="UnhandledError" Oct 30 18:11:01.812561 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:01.812352 6966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"korrel8r\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2928956623/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/51/5164b26f4cc06f20783d839dc18908e5f81e69881f12b11af759b9da340c3a0a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T181101Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4603776db8046df65b51c9f4298615ba89b42eff09c0828f1a8ac33b26bc6eb6®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=cluster-observability-operator----korrel8r-rhel9&akamai_signature=exp=1793384761~hmac=2e15b4396f04ee861b618660583b8f8a3c414e4fe9b2b1a8d39b675a8ead3301\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" Oct 30 18:11:02.138610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:02.138523 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.381233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:02.381196 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.620493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:02.620438 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:02.678664 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:02.678618 6966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"korrel8r\" with ImagePullBackOff: \"Back-off pulling image \\\"registry.redhat.io/cluster-observability-operator/korrel8r-rhel9@sha256:e3ebab37597e15fc58b989e7f3cf70f5561e96723f4f409333e8b99540386c8a\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: writing blob: storing blob to file \\\"/var/tmp/container_images_storage2928956623/1\\\": happened during read: unexpected EOF (while reconnecting: Get \\\"https://cdn01.quay.io/quayio-production-s3/sha256/51/5164b26f4cc06f20783d839dc18908e5f81e69881f12b11af759b9da340c3a0a?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIATAAF2YHTGR23ZTE6%2F20261030%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20261030T181101Z&X-Amz-Expires=600&X-Amz-SignedHeaders=host&X-Amz-Signature=4603776db8046df65b51c9f4298615ba89b42eff09c0828f1a8ac33b26bc6eb6®ion=us-east-1&namespace=redhat-prod&username=redhat-prod+registry_proxy&repo_name=cluster-observability-operator----korrel8r-rhel9&akamai_signature=exp=1793384761~hmac=2e15b4396f04ee861b618660583b8f8a3c414e4fe9b2b1a8d39b675a8ead3301\\\": EOF); artifact err: image reference: reference is a container image, not an OCI artifact\"" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" Oct 30 18:11:03.103982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.103933 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:03.304850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.304799 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:03.884307 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.884265 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:11:03.893646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.893622 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:03.907840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.907818 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:11:03.918890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.918865 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:11:03.919022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.918923 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:11:03.919022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.918932 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:11:03.919022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.918951 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:11:03.919022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.918950 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:11:03.919022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.918969 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-7brgw\"" Oct 30 18:11:03.919022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.919019 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:11:03.920650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.920633 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:11:03.988611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.988586 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:11:03.988672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:03.988638 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:11:04.008285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008249 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008329 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008386 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dzz2\" (UniqueName: \"kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008412 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008527 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008564 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008628 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.008767 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.008665 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.109846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.109808 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.109868 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.109905 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.109938 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.110019 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dzz2\" (UniqueName: \"kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.110058 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.110096 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.110123 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.110597 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.110687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.110629 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.111095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.111071 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.111208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.111192 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.113025 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.112997 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.113130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.113059 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.113130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.113113 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.126877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.126841 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:04.131684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.131665 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dzz2\" (UniqueName: \"kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:04.687922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:04.687887 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37d8712a-7364-4250-bc29-c6d0ae3d9938","Type":"ContainerStarted","Data":"0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1"} Oct 30 18:11:12.878654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:12.878624 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:12.905803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:12.905772 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:12.979328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:12.979298 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:12.979482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:12.979438 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:12.993006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:12.992986 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") DevicePath \"csi-f1d243cd1eff0987a64de94f5fb75fb003a2545b22342d982b5bcfa8f4a85c68\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:13.184095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:13.184059 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:13.208878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:13.208848 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:13.513291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:13.513202 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:13.849739 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:11:13.849714 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0baef175_fa16_40a7_bff3_a8ddaad769f5.slice/crio-aac5a056383a1c6321e8dafa534504e740bf9954ddb42a3fb556449d40ed8f99 WatchSource:0}: Error finding container aac5a056383a1c6321e8dafa534504e740bf9954ddb42a3fb556449d40ed8f99: Status 404 returned error can't find the container with id aac5a056383a1c6321e8dafa534504e740bf9954ddb42a3fb556449d40ed8f99 Oct 30 18:11:13.851692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:13.851672 6966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:11:13.856165 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:13.856111 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=10.975712929 podStartE2EDuration="14.856094257s" totalImagesPullingTime="3.880381328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:04.728573228 +0000 UTC m=+827.434758585" watchObservedRunningTime="2026-10-30 18:11:13.856094257 +0000 UTC m=+836.562279614" Oct 30 18:11:13.856677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:13.856659 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:11:14.737995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:14.737962 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerStarted","Data":"aac5a056383a1c6321e8dafa534504e740bf9954ddb42a3fb556449d40ed8f99"} Oct 30 18:11:17.198352 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:17.198256 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:11:17.755018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:17.754972 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerStarted","Data":"ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0"} Oct 30 18:11:18.121819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:18.121787 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:18.383661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:18.383585 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:19.767156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:19.767075 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerStarted","Data":"b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740"} Oct 30 18:11:19.768578 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:19.768553 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" event={"ID":"17ef22b9-8490-4f5c-8bc6-52b4516cb22e","Type":"ContainerStarted","Data":"6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c"} Oct 30 18:11:19.768774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:19.768758 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:11:19.809353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:19.809304 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" podStartSLOduration=1.776266128 podStartE2EDuration="21.809288395s" totalImagesPullingTime="20.033022267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=1 observedRunningTime="2026-10-30 18:11:19.807756455 +0000 UTC m=+842.513941813" watchObservedRunningTime="2026-10-30 18:11:19.809288395 +0000 UTC m=+842.515473753" Oct 30 18:11:21.780093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.780060 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerStarted","Data":"b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee"} Oct 30 18:11:21.780605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.780304 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:21.780605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.780331 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:21.780605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.780347 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:21.782881 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.782859 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:21.789329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.789299 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:21.849500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:21.849439 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=14.876325117 podStartE2EDuration="21.849427423s" totalImagesPullingTime="6.973102306s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:11:00 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:21.837335749 +0000 UTC m=+844.543521107" watchObservedRunningTime="2026-10-30 18:11:21.849427423 +0000 UTC m=+844.555612779" Oct 30 18:11:30.775970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:30.775939 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:11:33.797099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:33.797039 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:11:35.344406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.344372 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-nlftk"] Oct 30 18:11:35.344870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.344584 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" containerName="korrel8r" containerID="cri-o://6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c" gracePeriod=30 Oct 30 18:11:35.607206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.607154 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:11:35.663483 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.663440 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs\" (UniqueName: \"kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs\") pod \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " Oct 30 18:11:35.663680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.663564 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config\" (UniqueName: \"kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config\") pod \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " Oct 30 18:11:35.663680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.663589 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls\" (UniqueName: \"kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls\") pod \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\" (UID: \"17ef22b9-8490-4f5c-8bc6-52b4516cb22e\") " Oct 30 18:11:35.663940 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.663914 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config" pod "17ef22b9-8490-4f5c-8bc6-52b4516cb22e" (UID: "17ef22b9-8490-4f5c-8bc6-52b4516cb22e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:35.665877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.665847 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls" pod "17ef22b9-8490-4f5c-8bc6-52b4516cb22e" (UID: "17ef22b9-8490-4f5c-8bc6-52b4516cb22e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:35.665997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.665918 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs" pod "17ef22b9-8490-4f5c-8bc6-52b4516cb22e" (UID: "17ef22b9-8490-4f5c-8bc6-52b4516cb22e"). InnerVolumeSpecName "kube-api-access-p9mbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:35.750073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.750039 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:35.750348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.750255 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="37d8712a-7364-4250-bc29-c6d0ae3d9938" containerName="otc-container" containerID="cri-o://0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1" gracePeriod=30 Oct 30 18:11:35.767002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.766976 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9mbs\" (UniqueName: \"kubernetes.io/projected/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-kube-api-access-p9mbs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:35.767002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.767003 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:35.767134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.767012 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/17ef22b9-8490-4f5c-8bc6-52b4516cb22e-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:35.854240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.854207 6966 generic.go:356] "Generic (PLEG): container finished" podID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" containerID="6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c" exitCode=0 Oct 30 18:11:35.854399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.854264 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" Oct 30 18:11:35.854399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.854294 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" event={"ID":"17ef22b9-8490-4f5c-8bc6-52b4516cb22e","Type":"ContainerDied","Data":"6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c"} Oct 30 18:11:35.854399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.854359 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-nlftk" event={"ID":"17ef22b9-8490-4f5c-8bc6-52b4516cb22e","Type":"ContainerDied","Data":"95f2a840181ab4d86fca28e24cf7f42be1da47e3d42f21d5f65bcfa35ce50990"} Oct 30 18:11:35.854399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.854390 6966 scope.go:122] "RemoveContainer" containerID="6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c" Oct 30 18:11:35.863121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.863107 6966 scope.go:122] "RemoveContainer" containerID="6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c" Oct 30 18:11:35.863362 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:35.863348 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c\": container with ID starting with 6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c not found: ID does not exist" containerID="6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c" Oct 30 18:11:35.863415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.863369 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c"} err="failed to get container status \"6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c\": rpc error: code = NotFound desc = could not find container \"6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c\": container with ID starting with 6994f842a0acbb0185778c273fb98ee2357fb35fa9f4a7f80d5ba0dc5cbe473c not found: ID does not exist" Oct 30 18:11:35.889202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.889170 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:35.910494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.910277 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-nlftk"] Oct 30 18:11:35.918936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:35.912752 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-nlftk"] Oct 30 18:11:36.015897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.015872 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:36.070277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.070254 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7\" (UniqueName: \"kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7\") pod \"37d8712a-7364-4250-bc29-c6d0ae3d9938\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " Oct 30 18:11:36.070434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.070321 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal\" (UniqueName: \"kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal\") pod \"37d8712a-7364-4250-bc29-c6d0ae3d9938\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " Oct 30 18:11:36.070434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.070356 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs\") pod \"37d8712a-7364-4250-bc29-c6d0ae3d9938\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " Oct 30 18:11:36.070434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.070372 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs\") pod \"37d8712a-7364-4250-bc29-c6d0ae3d9938\" (UID: \"37d8712a-7364-4250-bc29-c6d0ae3d9938\") " Oct 30 18:11:36.070677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.070655 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal" pod "37d8712a-7364-4250-bc29-c6d0ae3d9938" (UID: "37d8712a-7364-4250-bc29-c6d0ae3d9938"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:36.072386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.072361 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs" pod "37d8712a-7364-4250-bc29-c6d0ae3d9938" (UID: "37d8712a-7364-4250-bc29-c6d0ae3d9938"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:36.072505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.072403 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs" pod "37d8712a-7364-4250-bc29-c6d0ae3d9938" (UID: "37d8712a-7364-4250-bc29-c6d0ae3d9938"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:36.072619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.072603 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7" pod "37d8712a-7364-4250-bc29-c6d0ae3d9938" (UID: "37d8712a-7364-4250-bc29-c6d0ae3d9938"). InnerVolumeSpecName "kube-api-access-7whf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:36.171760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.171695 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7whf7\" (UniqueName: \"kubernetes.io/projected/37d8712a-7364-4250-bc29-c6d0ae3d9938-kube-api-access-7whf7\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.171760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.171720 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/37d8712a-7364-4250-bc29-c6d0ae3d9938-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.171760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.171731 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.171760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.171739 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/37d8712a-7364-4250-bc29-c6d0ae3d9938-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:36.193969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.193935 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.255784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.255750 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:11:36.256012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.255986 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo" containerID="cri-o://ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0" gracePeriod=30 Oct 30 18:11:36.256114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.256027 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway-opa" containerID="cri-o://b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee" gracePeriod=30 Oct 30 18:11:36.256114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.256062 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway" containerID="cri-o://b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740" gracePeriod=30 Oct 30 18:11:36.257694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.257603 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:11:36.397891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.397857 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.648890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.648848 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:36.780937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.780863 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.38:8083/ready\": dial tcp 10.134.0.38:8083: connect: connection refused" Oct 30 18:11:36.784517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.784491 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:11:36.860694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.860670 6966 generic.go:356] "Generic (PLEG): container finished" podID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerID="b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee" exitCode=0 Oct 30 18:11:36.860694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.860689 6966 generic.go:356] "Generic (PLEG): container finished" podID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerID="ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0" exitCode=0 Oct 30 18:11:36.860819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.860739 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerDied","Data":"b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee"} Oct 30 18:11:36.860819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.860798 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerDied","Data":"ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0"} Oct 30 18:11:36.862937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.862919 6966 generic.go:356] "Generic (PLEG): container finished" podID="37d8712a-7364-4250-bc29-c6d0ae3d9938" containerID="0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1" exitCode=0 Oct 30 18:11:36.863017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.862968 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:36.863057 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.862967 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37d8712a-7364-4250-bc29-c6d0ae3d9938","Type":"ContainerDied","Data":"0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1"} Oct 30 18:11:36.863091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.863081 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"37d8712a-7364-4250-bc29-c6d0ae3d9938","Type":"ContainerDied","Data":"da451a71e5ad94853b6651669e54baf9817cd996a8c669e4f6eac69935be5916"} Oct 30 18:11:36.863124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.863111 6966 scope.go:122] "RemoveContainer" containerID="0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1" Oct 30 18:11:36.871629 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.871604 6966 scope.go:122] "RemoveContainer" containerID="0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1" Oct 30 18:11:36.871863 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:36.871846 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1\": container with ID starting with 0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1 not found: ID does not exist" containerID="0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1" Oct 30 18:11:36.871927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.871869 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1"} err="failed to get container status \"0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1\": rpc error: code = NotFound desc = could not find container \"0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1\": container with ID starting with 0d93249a345a80e66ffe2521f5b887b79348757a9264dd4aaa8af2feb7d16ab1 not found: ID does not exist" Oct 30 18:11:36.910768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.910747 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:36.915657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:36.915638 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:37.411308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.411281 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" path="/var/lib/kubelet/pods/17ef22b9-8490-4f5c-8bc6-52b4516cb22e/volumes" Oct 30 18:11:37.411678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.411664 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37d8712a-7364-4250-bc29-c6d0ae3d9938" path="/var/lib/kubelet/pods/37d8712a-7364-4250-bc29-c6d0ae3d9938/volumes" Oct 30 18:11:37.499993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.499969 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:37.583294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583223 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583264 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583294 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2\" (UniqueName: \"kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583347 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583371 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583430 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583531 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583557 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583587 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls\") pod \"0baef175-fa16-40a7-bff3-a8ddaad769f5\" (UID: \"0baef175-fa16-40a7-bff3-a8ddaad769f5\") " Oct 30 18:11:37.583904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583679 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:37.583904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583799 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:37.583904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583848 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:37.583904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583898 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.584112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.583918 6966 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.584536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.584501 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:37.585659 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.585624 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:37.585924 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.585906 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "kube-api-access-6dzz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:37.586263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.586240 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:37.586412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.586394 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:37.587539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.587520 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "0baef175-fa16-40a7-bff3-a8ddaad769f5" (UID: "0baef175-fa16-40a7-bff3-a8ddaad769f5"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:11:37.685209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685186 6966 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-gateway-tenants\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.685304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685224 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:11:37.685304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685238 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.685304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685248 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.685304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685257 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.685304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685267 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dzz2\" (UniqueName: \"kubernetes.io/projected/0baef175-fa16-40a7-bff3-a8ddaad769f5-kube-api-access-6dzz2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.685304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.685275 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/0baef175-fa16-40a7-bff3-a8ddaad769f5-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:37.694813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.694775 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:11:37.786054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.786028 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-f1d243cd1eff0987a64de94f5fb75fb003a2545b22342d982b5bcfa8f4a85c68\"" Oct 30 18:11:37.870836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.870777 6966 generic.go:356] "Generic (PLEG): container finished" podID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerID="b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740" exitCode=0 Oct 30 18:11:37.870836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.870825 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerDied","Data":"b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740"} Oct 30 18:11:37.870994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.870836 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:11:37.870994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.870874 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"0baef175-fa16-40a7-bff3-a8ddaad769f5","Type":"ContainerDied","Data":"aac5a056383a1c6321e8dafa534504e740bf9954ddb42a3fb556449d40ed8f99"} Oct 30 18:11:37.870994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.870902 6966 scope.go:122] "RemoveContainer" containerID="b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee" Oct 30 18:11:37.879943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.879922 6966 scope.go:122] "RemoveContainer" containerID="b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740" Oct 30 18:11:37.887802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.887787 6966 scope.go:122] "RemoveContainer" containerID="ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0" Oct 30 18:11:37.894992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.894977 6966 scope.go:122] "RemoveContainer" containerID="b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee" Oct 30 18:11:37.895242 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:37.895222 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee\": container with ID starting with b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee not found: ID does not exist" containerID="b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee" Oct 30 18:11:37.895281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.895250 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee"} err="failed to get container status \"b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee\": rpc error: code = NotFound desc = could not find container \"b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee\": container with ID starting with b679cd26b91d8c22a7ff43d3f2839ea62c4f64031614023356d4dc7cee0e73ee not found: ID does not exist" Oct 30 18:11:37.895281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.895265 6966 scope.go:122] "RemoveContainer" containerID="b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740" Oct 30 18:11:37.895522 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:37.895507 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740\": container with ID starting with b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740 not found: ID does not exist" containerID="b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740" Oct 30 18:11:37.895566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.895527 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740"} err="failed to get container status \"b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740\": rpc error: code = NotFound desc = could not find container \"b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740\": container with ID starting with b67a8d43fba2008692a97aa6b79c54fdd93bad407d9bf44e72ea4edda6452740 not found: ID does not exist" Oct 30 18:11:37.895566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.895540 6966 scope.go:122] "RemoveContainer" containerID="ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0" Oct 30 18:11:37.895785 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:37.895764 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0\": container with ID starting with ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0 not found: ID does not exist" containerID="ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0" Oct 30 18:11:37.895841 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.895794 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0"} err="failed to get container status \"ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0\": rpc error: code = NotFound desc = could not find container \"ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0\": container with ID starting with ecd92678e68df8ed2ef91482d72b5e4c3bd509f99a04433c429bf32dd2ab3de0 not found: ID does not exist" Oct 30 18:11:37.919907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.919887 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:11:37.930955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:37.930936 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:11:39.411187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:39.411155 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" path="/var/lib/kubelet/pods/0baef175-fa16-40a7-bff3-a8ddaad769f5/volumes" Oct 30 18:11:40.560333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560301 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-fg949"] Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560663 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway-opa" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560675 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway-opa" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560692 6966 cpu_manager.go:414] "containerMap: removing container" podUID="37d8712a-7364-4250-bc29-c6d0ae3d9938" containerName="otc-container" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560697 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37d8712a-7364-4250-bc29-c6d0ae3d9938" containerName="otc-container" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560707 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560713 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560723 6966 cpu_manager.go:414] "containerMap: removing container" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" containerName="korrel8r" Oct 30 18:11:40.560732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560728 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" containerName="korrel8r" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560740 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560744 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560812 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway-opa" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560823 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="37d8712a-7364-4250-bc29-c6d0ae3d9938" containerName="otc-container" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560829 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo-gateway" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560836 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ef22b9-8490-4f5c-8bc6-52b4516cb22e" containerName="korrel8r" Oct 30 18:11:40.561179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.560842 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0baef175-fa16-40a7-bff3-a8ddaad769f5" containerName="tempo" Oct 30 18:11:40.564413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.564391 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.578307 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.578276 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:11:40.589418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.589395 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:11:40.589582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.589537 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-qc47c\"" Oct 30 18:11:40.589644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.589602 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:11:40.589699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.589659 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:11:40.591374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.591350 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-fg949"] Oct 30 18:11:40.594006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.593979 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:40.607943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.607916 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.608045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.607950 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.608045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.607980 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdz8\" (UniqueName: \"kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.709070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.709046 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.709172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.709154 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.709216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.709197 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdz8\" (UniqueName: \"kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.709895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.709878 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.711472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.711440 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.731372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.731351 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdz8\" (UniqueName: \"kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8\") pod \"korrel8r-84f5b7859-fg949\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:40.875702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:40.875679 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:41.171767 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.171693 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-fg949"] Oct 30 18:11:41.176830 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:11:41.176792 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4added_7682_4622_816c_20b82b0f3117.slice/crio-0b1dd6c03427f35cbf468d178521f04bf2a1220da836a90f95ee35c6181a0c82 WatchSource:0}: Error finding container 0b1dd6c03427f35cbf468d178521f04bf2a1220da836a90f95ee35c6181a0c82: Status 404 returned error can't find the container with id 0b1dd6c03427f35cbf468d178521f04bf2a1220da836a90f95ee35c6181a0c82 Oct 30 18:11:41.541246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.541168 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:41.543964 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.543941 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.566524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.566497 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:41.575212 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.575178 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-b5f8018a\"" Oct 30 18:11:41.610157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.610127 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:11:41.610328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.610185 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-nldfn\"" Oct 30 18:11:41.610328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.610300 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:11:41.610439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.610405 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:11:41.616511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.616491 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.616621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.616546 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbkm\" (UniqueName: \"kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.616667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.616630 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.616706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.616692 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.616777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.616760 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.717190 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.717155 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.717342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.717230 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbkm\" (UniqueName: \"kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.717342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.717269 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.717342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.717328 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.717542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.717359 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.717542 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:41.717495 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:41.717653 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:41.717573 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert podName:72b84f7a-ab16-4ffd-a04f-01c995bba5f2 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:42.217551252 +0000 UTC m=+864.923736589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2") : secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:41.718143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.718114 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.719921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.719900 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.719987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.719924 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.741984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.741958 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbkm\" (UniqueName: \"kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:41.893701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.893670 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" event={"ID":"1e4added-7682-4622-816c-20b82b0f3117","Type":"ContainerStarted","Data":"b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77"} Oct 30 18:11:41.893857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.893727 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" event={"ID":"1e4added-7682-4622-816c-20b82b0f3117","Type":"ContainerStarted","Data":"0b1dd6c03427f35cbf468d178521f04bf2a1220da836a90f95ee35c6181a0c82"} Oct 30 18:11:41.893857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.893752 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:41.936641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:41.936596 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" podStartSLOduration=1.936583901 podStartE2EDuration="1.936583901s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:41.935841194 +0000 UTC m=+864.642026567" watchObservedRunningTime="2026-10-30 18:11:41.936583901 +0000 UTC m=+864.642769259" Oct 30 18:11:42.222093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:42.222018 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:42.222238 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:42.222140 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:42.222238 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:42.222192 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert podName:72b84f7a-ab16-4ffd-a04f-01c995bba5f2 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:43.222179457 +0000 UTC m=+865.928364791 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2") : secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:42.476749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:42.476676 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:43.231509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:43.231473 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:43.231954 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:43.231629 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-ta-client-cert: secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:43.231954 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:43.231691 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert podName:72b84f7a-ab16-4ffd-a04f-01c995bba5f2 nodeName:}" failed. No retries permitted until 2026-10-30 18:11:45.23167551 +0000 UTC m=+867.937860845 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "data-science-collector-ta-client-cert" (UniqueName: "kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert") pod "data-science-collector-collector-1" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2") : secret "data-science-collector-ta-client-cert" not found Oct 30 18:11:45.249526 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.249493 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:45.251808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.251780 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:45.458345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.458317 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:45.753228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.753198 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:45.758360 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:11:45.758332 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b84f7a_ab16_4ffd_a04f_01c995bba5f2.slice/crio-73dc5dcb16a078edba039cbb350bc1c057f62776fd6e819b3172676064d28211 WatchSource:0}: Error finding container 73dc5dcb16a078edba039cbb350bc1c057f62776fd6e819b3172676064d28211: Status 404 returned error can't find the container with id 73dc5dcb16a078edba039cbb350bc1c057f62776fd6e819b3172676064d28211 Oct 30 18:11:45.760650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.760612 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:11:45.760728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.760674 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:11:45.913761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.913727 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"72b84f7a-ab16-4ffd-a04f-01c995bba5f2","Type":"ContainerStarted","Data":"77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a"} Oct 30 18:11:45.913921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.913781 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"72b84f7a-ab16-4ffd-a04f-01c995bba5f2","Type":"ContainerStarted","Data":"73dc5dcb16a078edba039cbb350bc1c057f62776fd6e819b3172676064d28211"} Oct 30 18:11:45.955086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:45.955016 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=4.954999197 podStartE2EDuration="4.954999197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:45.953541152 +0000 UTC m=+868.659726506" watchObservedRunningTime="2026-10-30 18:11:45.954999197 +0000 UTC m=+868.661184554" Oct 30 18:11:47.199366 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:47.199333 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:11:52.900338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:52.900305 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:56.084267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.084229 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-fg949"] Oct 30 18:11:56.084977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.084418 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" podUID="1e4added-7682-4622-816c-20b82b0f3117" containerName="korrel8r" containerID="cri-o://b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77" gracePeriod=30 Oct 30 18:11:56.330217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.330186 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:56.436571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.436503 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8\" (UniqueName: \"kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8\") pod \"1e4added-7682-4622-816c-20b82b0f3117\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " Oct 30 18:11:56.436708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.436590 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config\" (UniqueName: \"kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config\") pod \"1e4added-7682-4622-816c-20b82b0f3117\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " Oct 30 18:11:56.436708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.436615 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls\" (UniqueName: \"kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls\") pod \"1e4added-7682-4622-816c-20b82b0f3117\" (UID: \"1e4added-7682-4622-816c-20b82b0f3117\") " Oct 30 18:11:56.437003 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.436976 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config" pod "1e4added-7682-4622-816c-20b82b0f3117" (UID: "1e4added-7682-4622-816c-20b82b0f3117"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:56.438420 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.438392 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8" pod "1e4added-7682-4622-816c-20b82b0f3117" (UID: "1e4added-7682-4622-816c-20b82b0f3117"). InnerVolumeSpecName "kube-api-access-kfdz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:56.438646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.438627 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls" pod "1e4added-7682-4622-816c-20b82b0f3117" (UID: "1e4added-7682-4622-816c-20b82b0f3117"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:56.537163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.537139 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e4added-7682-4622-816c-20b82b0f3117-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:56.537163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.537160 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e4added-7682-4622-816c-20b82b0f3117-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:56.537323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.537169 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfdz8\" (UniqueName: \"kubernetes.io/projected/1e4added-7682-4622-816c-20b82b0f3117-kube-api-access-kfdz8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:56.920877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.920840 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:56.921413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.921379 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="72b84f7a-ab16-4ffd-a04f-01c995bba5f2" containerName="otc-container" containerID="cri-o://77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a" gracePeriod=30 Oct 30 18:11:56.970261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.970221 6966 generic.go:356] "Generic (PLEG): container finished" podID="1e4added-7682-4622-816c-20b82b0f3117" containerID="b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77" exitCode=0 Oct 30 18:11:56.970396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.970280 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" Oct 30 18:11:56.970396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.970292 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" event={"ID":"1e4added-7682-4622-816c-20b82b0f3117","Type":"ContainerDied","Data":"b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77"} Oct 30 18:11:56.970396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.970355 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-84f5b7859-fg949" event={"ID":"1e4added-7682-4622-816c-20b82b0f3117","Type":"ContainerDied","Data":"0b1dd6c03427f35cbf468d178521f04bf2a1220da836a90f95ee35c6181a0c82"} Oct 30 18:11:56.970396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:56.970386 6966 scope.go:122] "RemoveContainer" containerID="b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77" Oct 30 18:11:57.031539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.031499 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-fg949"] Oct 30 18:11:57.037243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.037126 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-84f5b7859-fg949"] Oct 30 18:11:57.037325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.037289 6966 scope.go:122] "RemoveContainer" containerID="b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77" Oct 30 18:11:57.037641 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:57.037621 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77\": container with ID starting with b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77 not found: ID does not exist" containerID="b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77" Oct 30 18:11:57.037725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.037648 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77"} err="failed to get container status \"b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77\": rpc error: code = NotFound desc = could not find container \"b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77\": container with ID starting with b3116645b58009d49d8d262ec12e89008a0f14b19af6b459676c498130278e77 not found: ID does not exist" Oct 30 18:11:57.070856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.070814 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:57.166934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.166910 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:57.242917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.242844 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs\") pod \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " Oct 30 18:11:57.243052 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.242939 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") pod \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " Oct 30 18:11:57.243052 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.242985 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs\") pod \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " Oct 30 18:11:57.243163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.243067 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm\" (UniqueName: \"kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm\") pod \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " Oct 30 18:11:57.243163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.243104 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal\" (UniqueName: \"kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal\") pod \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\" (UID: \"72b84f7a-ab16-4ffd-a04f-01c995bba5f2\") " Oct 30 18:11:57.243616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.243584 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal" pod "72b84f7a-ab16-4ffd-a04f-01c995bba5f2" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:11:57.245215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.245188 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs" pod "72b84f7a-ab16-4ffd-a04f-01c995bba5f2" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:57.245443 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.245421 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert" pod "72b84f7a-ab16-4ffd-a04f-01c995bba5f2" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:57.245629 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.245573 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm" pod "72b84f7a-ab16-4ffd-a04f-01c995bba5f2" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2"). InnerVolumeSpecName "kube-api-access-nsbkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:11:57.245695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.245636 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs" pod "72b84f7a-ab16-4ffd-a04f-01c995bba5f2" (UID: "72b84f7a-ab16-4ffd-a04f-01c995bba5f2"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:11:57.312903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.312876 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:57.344314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.344291 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.344314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.344313 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.344445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.344326 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbkm\" (UniqueName: \"kubernetes.io/projected/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-kube-api-access-nsbkm\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.344445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.344334 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.344445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.344352 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/72b84f7a-ab16-4ffd-a04f-01c995bba5f2-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:11:57.411554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.411527 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4added-7682-4622-816c-20b82b0f3117" path="/var/lib/kubelet/pods/1e4added-7682-4622-816c-20b82b0f3117/volumes" Oct 30 18:11:57.517973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.517909 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:57.723914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.723882 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:11:57.976991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.976962 6966 generic.go:356] "Generic (PLEG): container finished" podID="72b84f7a-ab16-4ffd-a04f-01c995bba5f2" containerID="77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a" exitCode=0 Oct 30 18:11:57.977187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.977008 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:11:57.977187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.977052 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"72b84f7a-ab16-4ffd-a04f-01c995bba5f2","Type":"ContainerDied","Data":"77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a"} Oct 30 18:11:57.977187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.977099 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"72b84f7a-ab16-4ffd-a04f-01c995bba5f2","Type":"ContainerDied","Data":"73dc5dcb16a078edba039cbb350bc1c057f62776fd6e819b3172676064d28211"} Oct 30 18:11:57.977187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.977127 6966 scope.go:122] "RemoveContainer" containerID="77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a" Oct 30 18:11:57.985421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.985407 6966 scope.go:122] "RemoveContainer" containerID="77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a" Oct 30 18:11:57.985735 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:11:57.985713 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a\": container with ID starting with 77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a not found: ID does not exist" containerID="77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a" Oct 30 18:11:57.985814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:57.985742 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a"} err="failed to get container status \"77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a\": rpc error: code = NotFound desc = could not find container \"77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a\": container with ID starting with 77ea623ffb7f523ce6f5935f0832743f98add709290517c4737a042acbbdf65a not found: ID does not exist" Oct 30 18:11:58.024837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:58.024814 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:58.035971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:58.035951 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:11:59.411437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:11:59.411408 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72b84f7a-ab16-4ffd-a04f-01c995bba5f2" path="/var/lib/kubelet/pods/72b84f7a-ab16-4ffd-a04f-01c995bba5f2/volumes" Oct 30 18:12:00.428225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428194 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4"] Oct 30 18:12:00.428621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428521 6966 cpu_manager.go:414] "containerMap: removing container" podUID="72b84f7a-ab16-4ffd-a04f-01c995bba5f2" containerName="otc-container" Oct 30 18:12:00.428621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428533 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72b84f7a-ab16-4ffd-a04f-01c995bba5f2" containerName="otc-container" Oct 30 18:12:00.428621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428552 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1e4added-7682-4622-816c-20b82b0f3117" containerName="korrel8r" Oct 30 18:12:00.428621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428557 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4added-7682-4622-816c-20b82b0f3117" containerName="korrel8r" Oct 30 18:12:00.428621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428613 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4added-7682-4622-816c-20b82b0f3117" containerName="korrel8r" Oct 30 18:12:00.428621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.428624 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="72b84f7a-ab16-4ffd-a04f-01c995bba5f2" containerName="otc-container" Oct 30 18:12:00.436568 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.436543 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.462566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.462539 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:00.463272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.463252 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:12:00.463364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.463252 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:12:00.463364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.463293 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:12:00.463477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.463368 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-4nxjn\"" Oct 30 18:12:00.463477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.463389 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:12:00.464745 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.464724 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4"] Oct 30 18:12:00.568621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.568597 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xjh2\" (UniqueName: \"kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.568733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.568627 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.568733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.568646 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.669624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.669601 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xjh2\" (UniqueName: \"kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.669737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.669629 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.669737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.669648 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.670405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.670385 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.671715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.671691 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:00.672112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.672093 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.691866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.691816 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xjh2\" (UniqueName: \"kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2\") pod \"korrel8r-58bbdb5dcb-df6g4\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.748719 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.748694 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:00.875029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:00.874987 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:01.019525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.019441 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:01.022826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.022800 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.038928 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.038888 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:12:01.049989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.049965 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-5hqkj\"" Oct 30 18:12:01.050103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.050008 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:12:01.052134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.052114 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:01.058721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.058695 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4"] Oct 30 18:12:01.064912 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:01.064890 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f6bd8d9_4ac7_4ce8_9c4c_c9b85c365ac3.slice/crio-3eef70c3a8fdede971610301f092513af42d9ddb4fc3bd59c3d55837ea3cf712 WatchSource:0}: Error finding container 3eef70c3a8fdede971610301f092513af42d9ddb4fc3bd59c3d55837ea3cf712: Status 404 returned error can't find the container with id 3eef70c3a8fdede971610301f092513af42d9ddb4fc3bd59c3d55837ea3cf712 Oct 30 18:12:01.099452 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.099407 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:01.109146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.109130 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-c631c892\"" Oct 30 18:12:01.174534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.174509 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.174649 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.174541 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc86z\" (UniqueName: \"kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.174692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.174645 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.174692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.174678 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.275632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.275567 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.275632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.275603 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.275792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.275654 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.275792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.275674 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc86z\" (UniqueName: \"kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.276332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.276311 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.278011 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.277991 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.278082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.278032 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.296886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.296858 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc86z\" (UniqueName: \"kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z\") pod \"data-science-collector-collector-1\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.329143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.329115 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:01.334917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.334896 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:01.629692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.629664 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:01.634831 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:01.634803 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52f1198_2a65_465b_9def_7be7cfdba877.slice/crio-defeb956611ba1ef99e177b386f29b80b8d81ffc7a26c3c373201f2f41e3f84f WatchSource:0}: Error finding container defeb956611ba1ef99e177b386f29b80b8d81ffc7a26c3c373201f2f41e3f84f: Status 404 returned error can't find the container with id defeb956611ba1ef99e177b386f29b80b8d81ffc7a26c3c373201f2f41e3f84f Oct 30 18:12:01.636868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.636838 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:01.636931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:01.636895 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:02.000699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.000614 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" event={"ID":"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3","Type":"ContainerStarted","Data":"d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950"} Oct 30 18:12:02.000699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.000679 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" event={"ID":"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3","Type":"ContainerStarted","Data":"3eef70c3a8fdede971610301f092513af42d9ddb4fc3bd59c3d55837ea3cf712"} Oct 30 18:12:02.000902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.000711 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:02.002499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.002452 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f52f1198-2a65-465b-9def-7be7cfdba877","Type":"ContainerStarted","Data":"0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d"} Oct 30 18:12:02.002649 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.002521 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f52f1198-2a65-465b-9def-7be7cfdba877","Type":"ContainerStarted","Data":"defeb956611ba1ef99e177b386f29b80b8d81ffc7a26c3c373201f2f41e3f84f"} Oct 30 18:12:02.040028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.039985 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" podStartSLOduration=2.039972602 podStartE2EDuration="2.039972602s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:02.039156855 +0000 UTC m=+884.745342215" watchObservedRunningTime="2026-10-30 18:12:02.039972602 +0000 UTC m=+884.746157958" Oct 30 18:12:02.078802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:02.078759 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.07874661 podStartE2EDuration="2.07874661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:02.077854556 +0000 UTC m=+884.784039915" watchObservedRunningTime="2026-10-30 18:12:02.07874661 +0000 UTC m=+884.784931967" Oct 30 18:12:11.900994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:11.900960 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:13.009818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:13.009785 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:17.025923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.025891 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4"] Oct 30 18:12:17.026385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.026082 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" podUID="2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" containerName="korrel8r" containerID="cri-o://d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950" gracePeriod=30 Oct 30 18:12:17.203123 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:12:17.203046 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:12:17.270763 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.270741 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:17.299123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.299075 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls\" (UniqueName: \"kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls\") pod \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " Oct 30 18:12:17.299123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.299107 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config\" (UniqueName: \"kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config\") pod \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " Oct 30 18:12:17.299250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.299201 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2\" (UniqueName: \"kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2\") pod \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\" (UID: \"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3\") " Oct 30 18:12:17.299517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.299494 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config" pod "2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" (UID: "2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:17.301102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.301071 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls" pod "2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" (UID: "2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:17.301187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.301137 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2" pod "2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" (UID: "2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3"). InnerVolumeSpecName "kube-api-access-4xjh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:17.400821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.400755 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:17.401289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.401192 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.401289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.401249 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.401289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.401267 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xjh2\" (UniqueName: \"kubernetes.io/projected/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3-kube-api-access-4xjh2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.401289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.401065 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f52f1198-2a65-465b-9def-7be7cfdba877" containerName="otc-container" containerID="cri-o://0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d" gracePeriod=30 Oct 30 18:12:17.564972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.564937 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:17.631612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.631588 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:17.703403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.703379 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z\" (UniqueName: \"kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z\") pod \"f52f1198-2a65-465b-9def-7be7cfdba877\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " Oct 30 18:12:17.703580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.703423 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs\") pod \"f52f1198-2a65-465b-9def-7be7cfdba877\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " Oct 30 18:12:17.703580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.703530 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs\") pod \"f52f1198-2a65-465b-9def-7be7cfdba877\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " Oct 30 18:12:17.703580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.703570 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal\") pod \"f52f1198-2a65-465b-9def-7be7cfdba877\" (UID: \"f52f1198-2a65-465b-9def-7be7cfdba877\") " Oct 30 18:12:17.703961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.703935 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal" pod "f52f1198-2a65-465b-9def-7be7cfdba877" (UID: "f52f1198-2a65-465b-9def-7be7cfdba877"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:17.705650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.705626 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs" pod "f52f1198-2a65-465b-9def-7be7cfdba877" (UID: "f52f1198-2a65-465b-9def-7be7cfdba877"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:17.705770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.705757 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs" pod "f52f1198-2a65-465b-9def-7be7cfdba877" (UID: "f52f1198-2a65-465b-9def-7be7cfdba877"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:17.705941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.705920 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z" pod "f52f1198-2a65-465b-9def-7be7cfdba877" (UID: "f52f1198-2a65-465b-9def-7be7cfdba877"). InnerVolumeSpecName "kube-api-access-hc86z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:17.791586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.791551 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:17.804913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.804889 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f52f1198-2a65-465b-9def-7be7cfdba877-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.804913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.804911 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc86z\" (UniqueName: \"kubernetes.io/projected/f52f1198-2a65-465b-9def-7be7cfdba877-kube-api-access-hc86z\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.805132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.804922 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:17.805132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:17.804931 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f52f1198-2a65-465b-9def-7be7cfdba877-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:18.007432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.007346 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:18.081915 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.081887 6966 generic.go:356] "Generic (PLEG): container finished" podID="f52f1198-2a65-465b-9def-7be7cfdba877" containerID="0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d" exitCode=0 Oct 30 18:12:18.082302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.081946 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:18.082302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.081981 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f52f1198-2a65-465b-9def-7be7cfdba877","Type":"ContainerDied","Data":"0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d"} Oct 30 18:12:18.082302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.082075 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f52f1198-2a65-465b-9def-7be7cfdba877","Type":"ContainerDied","Data":"defeb956611ba1ef99e177b386f29b80b8d81ffc7a26c3c373201f2f41e3f84f"} Oct 30 18:12:18.082302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.082112 6966 scope.go:122] "RemoveContainer" containerID="0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d" Oct 30 18:12:18.083555 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.083534 6966 generic.go:356] "Generic (PLEG): container finished" podID="2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" containerID="d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950" exitCode=0 Oct 30 18:12:18.083650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.083570 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" event={"ID":"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3","Type":"ContainerDied","Data":"d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950"} Oct 30 18:12:18.083650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.083588 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" Oct 30 18:12:18.083650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.083609 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4" event={"ID":"2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3","Type":"ContainerDied","Data":"3eef70c3a8fdede971610301f092513af42d9ddb4fc3bd59c3d55837ea3cf712"} Oct 30 18:12:18.091648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.091632 6966 scope.go:122] "RemoveContainer" containerID="0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d" Oct 30 18:12:18.091889 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:12:18.091873 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d\": container with ID starting with 0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d not found: ID does not exist" containerID="0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d" Oct 30 18:12:18.091932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.091895 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d"} err="failed to get container status \"0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d\": rpc error: code = NotFound desc = could not find container \"0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d\": container with ID starting with 0574a1ae5339ca5510cfb809a231680f816e5b79a37f686e47b6ec0a2a07312d not found: ID does not exist" Oct 30 18:12:18.091932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.091909 6966 scope.go:122] "RemoveContainer" containerID="d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950" Oct 30 18:12:18.099297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.099279 6966 scope.go:122] "RemoveContainer" containerID="d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950" Oct 30 18:12:18.099583 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:12:18.099566 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950\": container with ID starting with d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950 not found: ID does not exist" containerID="d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950" Oct 30 18:12:18.099643 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.099590 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950"} err="failed to get container status \"d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950\": rpc error: code = NotFound desc = could not find container \"d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950\": container with ID starting with d34a8a37a9b687bd4d1c1edd77ac0a2cd8ddbbd84756ace462f37b70cefbf950 not found: ID does not exist" Oct 30 18:12:18.128937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.128914 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:18.139857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.139837 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:18.165229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.165207 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4"] Oct 30 18:12:18.169624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.169606 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-58bbdb5dcb-df6g4"] Oct 30 18:12:18.244413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:18.244394 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:19.410862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:19.410828 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" path="/var/lib/kubelet/pods/2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3/volumes" Oct 30 18:12:19.411392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:19.411368 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52f1198-2a65-465b-9def-7be7cfdba877" path="/var/lib/kubelet/pods/f52f1198-2a65-465b-9def-7be7cfdba877/volumes" Oct 30 18:12:20.491189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491153 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx"] Oct 30 18:12:20.491572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491498 6966 cpu_manager.go:414] "containerMap: removing container" podUID="2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" containerName="korrel8r" Oct 30 18:12:20.491572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491510 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" containerName="korrel8r" Oct 30 18:12:20.491572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491536 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f52f1198-2a65-465b-9def-7be7cfdba877" containerName="otc-container" Oct 30 18:12:20.491572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491541 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52f1198-2a65-465b-9def-7be7cfdba877" containerName="otc-container" Oct 30 18:12:20.491705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491596 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f6bd8d9-4ac7-4ce8-9c4c-c9b85c365ac3" containerName="korrel8r" Oct 30 18:12:20.491705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.491608 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52f1198-2a65-465b-9def-7be7cfdba877" containerName="otc-container" Oct 30 18:12:20.494804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.494781 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.517140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.517107 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:12:20.520285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.520158 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:12:20.520285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.520187 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-v8sqd\"" Oct 30 18:12:20.520285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.520207 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:12:20.520285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.520261 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:12:20.522424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.522403 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx"] Oct 30 18:12:20.527094 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.527075 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.527202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.527112 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.527202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.527149 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr26p\" (UniqueName: \"kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.547931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.547906 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:20.628370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.628351 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.628514 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.628388 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.628514 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.628450 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr26p\" (UniqueName: \"kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.629170 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.629153 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.630815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.630792 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.650773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.650750 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr26p\" (UniqueName: \"kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p\") pod \"korrel8r-5c7449969d-cj9wx\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:20.775157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.775087 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:20.806945 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:20.806922 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:21.102418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.102386 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx"] Oct 30 18:12:21.107551 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:21.107523 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fa62db7_b24c_4d27_b3bd_4be6c55fbaf8.slice/crio-f059546dda3e95905cac047d43a54afac3688f31084b91807380cd9b1824f167 WatchSource:0}: Error finding container f059546dda3e95905cac047d43a54afac3688f31084b91807380cd9b1824f167: Status 404 returned error can't find the container with id f059546dda3e95905cac047d43a54afac3688f31084b91807380cd9b1824f167 Oct 30 18:12:21.562152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.562131 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:21.565210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.565188 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.594792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.594766 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:21.600664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.600625 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:12:21.600807 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.600683 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:12:21.604137 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.604120 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-kskgg\"" Oct 30 18:12:21.604214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.604187 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:12:21.637679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.637652 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghrgg\" (UniqueName: \"kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.637770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.637697 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.637770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.637740 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.637843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.637775 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.637843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.637805 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.656661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.656643 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-4d267ad2\"" Oct 30 18:12:21.664006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.663984 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:21.738290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.738263 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.738423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.738306 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.738423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.738343 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.738423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.738390 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghrgg\" (UniqueName: \"kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.738423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.738419 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.739222 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.739102 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.741238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.741221 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.741324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.741232 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.741324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.741295 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.760231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.760205 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghrgg\" (UniqueName: \"kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg\") pod \"data-science-collector-collector-1\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.878973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.878912 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:21.888031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:21.888003 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:22.106335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.106298 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" event={"ID":"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8","Type":"ContainerStarted","Data":"2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa"} Oct 30 18:12:22.106532 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.106360 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:22.106532 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.106384 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" event={"ID":"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8","Type":"ContainerStarted","Data":"f059546dda3e95905cac047d43a54afac3688f31084b91807380cd9b1824f167"} Oct 30 18:12:22.145002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.144926 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" podStartSLOduration=2.144913211 podStartE2EDuration="2.144913211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:22.144324687 +0000 UTC m=+904.850510054" watchObservedRunningTime="2026-10-30 18:12:22.144913211 +0000 UTC m=+904.851098568" Oct 30 18:12:22.198802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.198770 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:22.203448 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:22.203418 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbb7392_8787_4565_8dc8_fbd83bc49dbb.slice/crio-949dc85ffd725595dcfa74cf50cef5b4e5248eb7439f55595276ac11e0f463da WatchSource:0}: Error finding container 949dc85ffd725595dcfa74cf50cef5b4e5248eb7439f55595276ac11e0f463da: Status 404 returned error can't find the container with id 949dc85ffd725595dcfa74cf50cef5b4e5248eb7439f55595276ac11e0f463da Oct 30 18:12:22.205552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.205515 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:22.205632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:22.205576 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:23.112881 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:23.112839 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bbb7392-8787-4565-8dc8-fbd83bc49dbb","Type":"ContainerStarted","Data":"b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8"} Oct 30 18:12:23.113291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:23.112924 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bbb7392-8787-4565-8dc8-fbd83bc49dbb","Type":"ContainerStarted","Data":"949dc85ffd725595dcfa74cf50cef5b4e5248eb7439f55595276ac11e0f463da"} Oct 30 18:12:23.153024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:23.152979 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.152967549 podStartE2EDuration="2.152967549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:23.151839504 +0000 UTC m=+905.858024862" watchObservedRunningTime="2026-10-30 18:12:23.152967549 +0000 UTC m=+905.859152905" Oct 30 18:12:33.115175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:33.115090 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:36.306870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.306835 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx"] Oct 30 18:12:36.307696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.307663 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" podUID="9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" containerName="korrel8r" containerID="cri-o://2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa" gracePeriod=30 Oct 30 18:12:36.547704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.547679 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:36.648379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.648347 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls\" (UniqueName: \"kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls\") pod \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " Oct 30 18:12:36.648582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.648387 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p\" (UniqueName: \"kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p\") pod \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " Oct 30 18:12:36.648582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.648534 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config\" (UniqueName: \"kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config\") pod \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\" (UID: \"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8\") " Oct 30 18:12:36.648917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.648888 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config" pod "9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" (UID: "9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:36.650407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.650382 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls" pod "9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" (UID: "9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:36.650525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.650437 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p" pod "9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" (UID: "9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8"). InnerVolumeSpecName "kube-api-access-zr26p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:36.749819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.749795 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:36.749819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.749819 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr26p\" (UniqueName: \"kubernetes.io/projected/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-kube-api-access-zr26p\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:36.749965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:36.749831 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.182238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.182206 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:37.182432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.182411 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="0bbb7392-8787-4565-8dc8-fbd83bc49dbb" containerName="otc-container" containerID="cri-o://b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8" gracePeriod=30 Oct 30 18:12:37.185826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.185789 6966 generic.go:356] "Generic (PLEG): container finished" podID="9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" containerID="2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa" exitCode=0 Oct 30 18:12:37.185935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.185850 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" Oct 30 18:12:37.185935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.185865 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" event={"ID":"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8","Type":"ContainerDied","Data":"2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa"} Oct 30 18:12:37.185935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.185931 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx" event={"ID":"9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8","Type":"ContainerDied","Data":"f059546dda3e95905cac047d43a54afac3688f31084b91807380cd9b1824f167"} Oct 30 18:12:37.186105 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.185955 6966 scope.go:122] "RemoveContainer" containerID="2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa" Oct 30 18:12:37.199075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.198810 6966 scope.go:122] "RemoveContainer" containerID="2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa" Oct 30 18:12:37.199201 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:12:37.199137 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa\": container with ID starting with 2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa not found: ID does not exist" containerID="2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa" Oct 30 18:12:37.199201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.199181 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa"} err="failed to get container status \"2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa\": rpc error: code = NotFound desc = could not find container \"2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa\": container with ID starting with 2fef804160a8949887a1692a2603a7d0a745b784045555f782ba9188139daefa not found: ID does not exist" Oct 30 18:12:37.297979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.297952 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx"] Oct 30 18:12:37.314251 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.314226 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5c7449969d-cj9wx"] Oct 30 18:12:37.335314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.335284 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:37.412490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.412203 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" path="/var/lib/kubelet/pods/9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8/volumes" Oct 30 18:12:37.429958 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.429939 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:37.558320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.558243 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal\") pod \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " Oct 30 18:12:37.558320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.558295 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs\") pod \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " Oct 30 18:12:37.558559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.558335 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs\") pod \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " Oct 30 18:12:37.558559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.558357 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert\") pod \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " Oct 30 18:12:37.558559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.558426 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg\" (UniqueName: \"kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg\") pod \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\" (UID: \"0bbb7392-8787-4565-8dc8-fbd83bc49dbb\") " Oct 30 18:12:37.558745 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.558658 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal" pod "0bbb7392-8787-4565-8dc8-fbd83bc49dbb" (UID: "0bbb7392-8787-4565-8dc8-fbd83bc49dbb"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:37.560422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.560385 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs" pod "0bbb7392-8787-4565-8dc8-fbd83bc49dbb" (UID: "0bbb7392-8787-4565-8dc8-fbd83bc49dbb"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:37.560670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.560644 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg" pod "0bbb7392-8787-4565-8dc8-fbd83bc49dbb" (UID: "0bbb7392-8787-4565-8dc8-fbd83bc49dbb"). InnerVolumeSpecName "kube-api-access-ghrgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:37.560797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.560739 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert" pod "0bbb7392-8787-4565-8dc8-fbd83bc49dbb" (UID: "0bbb7392-8787-4565-8dc8-fbd83bc49dbb"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:37.560895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.560873 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs" pod "0bbb7392-8787-4565-8dc8-fbd83bc49dbb" (UID: "0bbb7392-8787-4565-8dc8-fbd83bc49dbb"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:37.572617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.572589 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:37.659304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.659278 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.659304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.659300 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.659451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.659311 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghrgg\" (UniqueName: \"kubernetes.io/projected/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-kube-api-access-ghrgg\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.659451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.659321 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.659451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.659329 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0bbb7392-8787-4565-8dc8-fbd83bc49dbb-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:12:37.811665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:37.811631 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:38.031302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.031270 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:38.191827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.191747 6966 generic.go:356] "Generic (PLEG): container finished" podID="0bbb7392-8787-4565-8dc8-fbd83bc49dbb" containerID="b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8" exitCode=0 Oct 30 18:12:38.191827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.191795 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:38.192056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.191831 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bbb7392-8787-4565-8dc8-fbd83bc49dbb","Type":"ContainerDied","Data":"b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8"} Oct 30 18:12:38.192056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.191882 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"0bbb7392-8787-4565-8dc8-fbd83bc49dbb","Type":"ContainerDied","Data":"949dc85ffd725595dcfa74cf50cef5b4e5248eb7439f55595276ac11e0f463da"} Oct 30 18:12:38.192056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.191899 6966 scope.go:122] "RemoveContainer" containerID="b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8" Oct 30 18:12:38.200557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.200542 6966 scope.go:122] "RemoveContainer" containerID="b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8" Oct 30 18:12:38.200790 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:12:38.200773 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8\": container with ID starting with b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8 not found: ID does not exist" containerID="b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8" Oct 30 18:12:38.200831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.200799 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8"} err="failed to get container status \"b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8\": rpc error: code = NotFound desc = could not find container \"b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8\": container with ID starting with b956d75b9f294301257bf56dbe40896bb5f3837019ccbc93e62ca938ba4ac1e8 not found: ID does not exist" Oct 30 18:12:38.239394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.239370 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:38.250345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:38.250324 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:39.411499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:39.411454 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbb7392-8787-4565-8dc8-fbd83bc49dbb" path="/var/lib/kubelet/pods/0bbb7392-8787-4565-8dc8-fbd83bc49dbb/volumes" Oct 30 18:12:44.495728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:44.495689 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:46.207715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.207663 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:46.429884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.429854 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp"] Oct 30 18:12:46.430179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.430168 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0bbb7392-8787-4565-8dc8-fbd83bc49dbb" containerName="otc-container" Oct 30 18:12:46.430228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.430180 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbb7392-8787-4565-8dc8-fbd83bc49dbb" containerName="otc-container" Oct 30 18:12:46.430228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.430212 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" containerName="korrel8r" Oct 30 18:12:46.430228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.430218 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" containerName="korrel8r" Oct 30 18:12:46.430316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.430281 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fa62db7-b24c-4d27-b3bd-4be6c55fbaf8" containerName="korrel8r" Oct 30 18:12:46.430316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.430292 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbb7392-8787-4565-8dc8-fbd83bc49dbb" containerName="otc-container" Oct 30 18:12:46.433638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.433616 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.442105 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.442071 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:46.458287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.458238 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp"] Oct 30 18:12:46.471538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.471515 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh"] Oct 30 18:12:46.474286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.474269 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.496413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.496387 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh"] Oct 30 18:12:46.508615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.508589 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:12:46.508804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.508781 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:12:46.515209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.508770 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:12:46.515575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.515539 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:12:46.515686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.515576 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:12:46.515753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.515736 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:12:46.515903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.515883 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:12:46.516095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.515968 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-9ccnv\"" Oct 30 18:12:46.516207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.516034 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:12:46.516332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.516314 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-7wz4x\"" Oct 30 18:12:46.517798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.517783 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c"] Oct 30 18:12:46.520259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.520241 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.528305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528279 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggpmk\" (UniqueName: \"kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.528409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528308 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.528409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528328 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.528409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528342 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.528409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528399 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.528658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528490 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.528658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528541 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.528658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.528600 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lbhx\" (UniqueName: \"kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.537595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.537578 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c"] Oct 30 18:12:46.609767 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.609747 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-b5w9h\"" Oct 30 18:12:46.609885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.609831 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:12:46.609885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.609838 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:12:46.629168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629146 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggpmk\" (UniqueName: \"kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.629256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629173 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.629256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629194 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.629256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629210 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.629256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629235 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.629426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629261 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.629426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629335 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.629426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629377 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.629426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629404 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.629670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629527 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.629670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629604 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lbhx\" (UniqueName: \"kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.629965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.629932 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.630061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.630002 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.630122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.630082 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.630238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.630220 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.631727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.631708 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.631921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.631901 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.651069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.651051 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggpmk\" (UniqueName: \"kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-mppnp\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.651347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.651325 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lbhx\" (UniqueName: \"kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx\") pod \"data-science-thanos-querier-proxy-6998ff8b98-4j6nh\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.659646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.659623 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:46.730754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.730705 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.730754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.730732 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.730867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.730799 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.731390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.731370 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.732803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.732783 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.748707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.748690 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:12:46.753475 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.753441 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984\") pod \"korrel8r-5cbcc89b8f-hck8c\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.788673 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.788644 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:12:46.832016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.831785 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:46.879989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:46.879949 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:47.043964 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.043894 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp"] Oct 30 18:12:47.048543 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:47.048511 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda097e0c7_aa30_4e0a_97ed_1c402da5a3d6.slice/crio-d1fd8b8d79ef3d823b6cacd0a49a8e617485c458c16787db90e36622dee906fe WatchSource:0}: Error finding container d1fd8b8d79ef3d823b6cacd0a49a8e617485c458c16787db90e36622dee906fe: Status 404 returned error can't find the container with id d1fd8b8d79ef3d823b6cacd0a49a8e617485c458c16787db90e36622dee906fe Oct 30 18:12:47.093308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.093282 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh"] Oct 30 18:12:47.097710 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:47.097682 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ccb131f_d605_44f8_9f46_c86e995fce69.slice/crio-1ce185e825e612e3105b94bdd536f7c2e3f67f189b8f6ef3e3e19d3ffb749122 WatchSource:0}: Error finding container 1ce185e825e612e3105b94bdd536f7c2e3f67f189b8f6ef3e3e19d3ffb749122: Status 404 returned error can't find the container with id 1ce185e825e612e3105b94bdd536f7c2e3f67f189b8f6ef3e3e19d3ffb749122 Oct 30 18:12:47.118226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.118199 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:47.134388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.134364 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c"] Oct 30 18:12:47.138640 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:47.138617 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10981b78_93cd_46b1_86ee_e648fd600bdb.slice/crio-f1b5be5dd403d842c07d176bcdd9319ee877379d595e6b19c66caad6b83cf316 WatchSource:0}: Error finding container f1b5be5dd403d842c07d176bcdd9319ee877379d595e6b19c66caad6b83cf316: Status 404 returned error can't find the container with id f1b5be5dd403d842c07d176bcdd9319ee877379d595e6b19c66caad6b83cf316 Oct 30 18:12:47.202179 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:12:47.202152 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:12:47.244512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.244487 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerStarted","Data":"1ce185e825e612e3105b94bdd536f7c2e3f67f189b8f6ef3e3e19d3ffb749122"} Oct 30 18:12:47.248344 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.248311 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" event={"ID":"10981b78-93cd-46b1-86ee-e648fd600bdb","Type":"ContainerStarted","Data":"ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8"} Oct 30 18:12:47.248508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.248365 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" event={"ID":"10981b78-93cd-46b1-86ee-e648fd600bdb","Type":"ContainerStarted","Data":"f1b5be5dd403d842c07d176bcdd9319ee877379d595e6b19c66caad6b83cf316"} Oct 30 18:12:47.248508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.248420 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:47.249606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.249581 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerStarted","Data":"d1fd8b8d79ef3d823b6cacd0a49a8e617485c458c16787db90e36622dee906fe"} Oct 30 18:12:47.286834 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.286798 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" podStartSLOduration=1.286788609 podStartE2EDuration="1.286788609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:47.286441988 +0000 UTC m=+929.992627344" watchObservedRunningTime="2026-10-30 18:12:47.286788609 +0000 UTC m=+929.992973967" Oct 30 18:12:47.331880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.331857 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:47.571315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.571280 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:47.978404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.978370 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:47.981760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:47.981735 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.011834 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.009841 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:48.051540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.051505 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.051697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.051551 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2hzh\" (UniqueName: \"kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.051697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.051587 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.051697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.051625 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.051880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.051713 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.051880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.051761 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.060715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.060682 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-wkhjr\"" Oct 30 18:12:48.060879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.060720 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:12:48.060879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.060753 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:12:48.060879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.060812 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:12:48.060879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.060871 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:12:48.061069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.060899 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:12:48.070773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.070741 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:48.152636 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.152600 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.153100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.153080 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.153217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.153132 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.153217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.153185 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.153217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.153209 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2hzh\" (UniqueName: \"kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.153402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.153256 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.154254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.154220 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.156879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.156858 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.159395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.159374 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.161008 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.160990 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.163195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.163180 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.178639 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.178584 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2hzh\" (UniqueName: \"kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh\") pod \"data-science-collector-collector-1\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.302402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.302331 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:12:48.303999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.303928 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:48.622787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.622752 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:12:48.630238 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:48.630208 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f730bb1_7d33_4e72_a0f6_240280561cd5.slice/crio-5830527d160480c2d4bcb6b511bb87b32d95b4e3b834559033e24e51470c33e2 WatchSource:0}: Error finding container 5830527d160480c2d4bcb6b511bb87b32d95b4e3b834559033e24e51470c33e2: Status 404 returned error can't find the container with id 5830527d160480c2d4bcb6b511bb87b32d95b4e3b834559033e24e51470c33e2 Oct 30 18:12:48.633349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.632985 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:48.633349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:48.633089 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:12:49.273070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:49.273023 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f730bb1-7d33-4e72-a0f6-240280561cd5","Type":"ContainerStarted","Data":"6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6"} Oct 30 18:12:49.273346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:49.273089 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f730bb1-7d33-4e72-a0f6-240280561cd5","Type":"ContainerStarted","Data":"5830527d160480c2d4bcb6b511bb87b32d95b4e3b834559033e24e51470c33e2"} Oct 30 18:12:50.280476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:50.280373 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerStarted","Data":"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5"} Oct 30 18:12:50.280873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:50.280488 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerStarted","Data":"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8"} Oct 30 18:12:50.282324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:50.282296 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerStarted","Data":"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e"} Oct 30 18:12:50.282479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:50.282347 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerStarted","Data":"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8"} Oct 30 18:12:51.289551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:51.289516 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerStarted","Data":"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6"} Oct 30 18:12:51.291506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:51.291477 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerStarted","Data":"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616"} Oct 30 18:12:51.331150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:51.331060 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=4.331044615 podStartE2EDuration="4.331044615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:49.315515178 +0000 UTC m=+932.021700534" watchObservedRunningTime="2026-10-30 18:12:51.331044615 +0000 UTC m=+934.037229972" Oct 30 18:12:51.331657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:51.331615 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" podStartSLOduration=1.505144099 podStartE2EDuration="5.331606033s" totalImagesPullingTime="3.826461934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:51.330255147 +0000 UTC m=+934.036440505" watchObservedRunningTime="2026-10-30 18:12:51.331606033 +0000 UTC m=+934.037791391" Oct 30 18:12:51.373336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:51.373296 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" podStartSLOduration=1.469263485 podStartE2EDuration="5.373285144s" totalImagesPullingTime="3.904021659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:46 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:51.372695673 +0000 UTC m=+934.078881031" watchObservedRunningTime="2026-10-30 18:12:51.373285144 +0000 UTC m=+934.079470501" Oct 30 18:12:55.494972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:55.494935 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:55.737080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:55.737045 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:55.956724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:55.956690 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:56.014184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.014151 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:12:56.019621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.019598 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.047473 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.047444 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:12:56.102697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.102672 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:12:56.102697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.102675 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:12:56.102697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.102675 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:12:56.102967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.102710 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:12:56.102967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.102894 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-fjh62\"" Oct 30 18:12:56.123238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123214 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.123357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123286 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dss9x\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.123406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123356 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.123492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123429 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.123562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123500 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.123620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123581 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.123672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.123654 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224546 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dss9x\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224590 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224618 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224661 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224692 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224740 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.224895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.224773 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.225298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.225227 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.227434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.227389 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.227688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.227663 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.227756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.227682 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.227756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.227731 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.227859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.227792 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.246391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.246365 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dss9x\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.247493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.247469 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:56.339726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.339702 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:12:56.498722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.498647 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:56.667783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:56.667746 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:12:56.672252 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:12:56.672225 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aa3074f_51be_4249_80b7_255fbd4b3205.slice/crio-562f01412d7d0a9fc6d45ae163eae39451d3b44401111f251304e690552427e6 WatchSource:0}: Error finding container 562f01412d7d0a9fc6d45ae163eae39451d3b44401111f251304e690552427e6: Status 404 returned error can't find the container with id 562f01412d7d0a9fc6d45ae163eae39451d3b44401111f251304e690552427e6 Oct 30 18:12:57.322740 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:57.322699 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerStarted","Data":"562f01412d7d0a9fc6d45ae163eae39451d3b44401111f251304e690552427e6"} Oct 30 18:12:58.266495 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:58.266417 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:12:58.542001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:58.541940 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:58.762309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:58.762278 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:59.199360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.199331 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:12:59.203324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.203303 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.233353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.233324 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:12:59.260287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.260263 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:12:59.260710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.260238 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.263728 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.263790 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.263863 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-tdqmt\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.263910 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.264092 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.264176 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.264212 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:12:59.267017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.264263 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:12:59.335822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.335797 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerStarted","Data":"66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf"} Oct 30 18:12:59.353005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.352981 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353020 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353052 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353172 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353232 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxj6h\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353269 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353298 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353352 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353375 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353473 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353441 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353545 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.353601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.353573 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.410787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.410762 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:12:59.454427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454368 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454399 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454430 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454531 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454561 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454610 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454643 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454677 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454709 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.454825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454768 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxj6h\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.455034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454831 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.455034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.454861 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.455868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.455796 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.455977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.455867 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.456113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.456090 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.458256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458203 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.458387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458363 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.458518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458390 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.458518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458404 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.458811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458791 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.458982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458958 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.459019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.458985 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.459687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.459667 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:12:59.477994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:12:59.477973 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxj6h\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:02.360421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:02.360389 6966 generic.go:356] "Generic (PLEG): container finished" podID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerID="66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf" exitCode=0 Oct 30 18:13:02.360849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:02.360449 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerDied","Data":"66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf"} Oct 30 18:13:04.375755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:04.375723 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerStarted","Data":"a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0"} Oct 30 18:13:05.382728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.382697 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerStarted","Data":"254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676"} Oct 30 18:13:05.383245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.382844 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:13:05.451780 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.451732 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.836802869 podStartE2EDuration="10.451719433s" totalImagesPullingTime="2.92834427s" totalInitContainerRuntime="3.686572294s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:55 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:05.439692657 +0000 UTC m=+948.145878014" watchObservedRunningTime="2026-10-30 18:13:05.451719433 +0000 UTC m=+948.157904789" Oct 30 18:13:05.707696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.707620 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:05.733981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.733949 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:05.808982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.808954 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:05.809100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.809080 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:05.822669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:05.822651 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") DevicePath \"csi-83df9dfad9fd2ac35dca1441af096f35871cdbdee4927aba3c5e3091d60e99b5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:06.000105 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:06.000077 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:06.025970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:06.025936 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:06.123564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:06.123539 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:06.461950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:06.461923 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:13:06.466433 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:13:06.466402 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96695215_60d4_43b7_891a_130ee4cf7055.slice/crio-0c9d4f8f4d7afd550b44947249c42a0e5900fa57a56e6e108803be2cb418e4ba WatchSource:0}: Error finding container 0c9d4f8f4d7afd550b44947249c42a0e5900fa57a56e6e108803be2cb418e4ba: Status 404 returned error can't find the container with id 0c9d4f8f4d7afd550b44947249c42a0e5900fa57a56e6e108803be2cb418e4ba Oct 30 18:13:07.394785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:07.394747 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerStarted","Data":"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5"} Oct 30 18:13:07.394950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:07.394806 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerStarted","Data":"0c9d4f8f4d7afd550b44947249c42a0e5900fa57a56e6e108803be2cb418e4ba"} Oct 30 18:13:11.389596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:11.389565 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:13:11.417468 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:11.417439 6966 generic.go:356] "Generic (PLEG): container finished" podID="96695215-60d4-43b7-891a-130ee4cf7055" containerID="a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5" exitCode=0 Oct 30 18:13:11.417618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:11.417514 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerDied","Data":"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5"} Oct 30 18:13:15.440981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:15.440944 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerStarted","Data":"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155"} Oct 30 18:13:15.441368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:15.441013 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerStarted","Data":"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52"} Oct 30 18:13:16.109788 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:16.109748 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:17.197940 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:17.197906 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:13:17.454864 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:17.454794 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerStarted","Data":"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8"} Oct 30 18:13:17.540036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:17.539965 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=12.80121624 podStartE2EDuration="21.539948932s" totalImagesPullingTime="5.113484301s" totalInitContainerRuntime="3.625248391s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:12:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:17.526767061 +0000 UTC m=+960.232952418" watchObservedRunningTime="2026-10-30 18:13:17.539948932 +0000 UTC m=+960.246134291" Oct 30 18:13:18.389992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:18.389950 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:21.123939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:21.123905 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:21.124410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:21.123985 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:21.129760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:21.129736 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:21.481111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:21.481037 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:26.663180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.663146 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp"] Oct 30 18:13:26.663656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.663369 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="kube-rbac-proxy" containerID="cri-o://88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" gracePeriod=30 Oct 30 18:13:26.663656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.663446 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="prom-label-proxy" containerID="cri-o://8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" gracePeriod=30 Oct 30 18:13:26.663656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.663528 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="method-gate" containerID="cri-o://6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" gracePeriod=30 Oct 30 18:13:26.681539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.681511 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh"] Oct 30 18:13:26.681770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.681744 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="kube-rbac-proxy" containerID="cri-o://0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" gracePeriod=30 Oct 30 18:13:26.681972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.681765 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="prom-label-proxy" containerID="cri-o://f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" gracePeriod=30 Oct 30 18:13:26.682082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.681834 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="method-gate" containerID="cri-o://840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" gracePeriod=30 Oct 30 18:13:26.704755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.704732 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c"] Oct 30 18:13:26.705108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.705037 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" podUID="10981b78-93cd-46b1-86ee-e648fd600bdb" containerName="korrel8r" containerID="cri-o://ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8" gracePeriod=30 Oct 30 18:13:26.961026 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.960998 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:13:26.998380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:26.998353 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:13:27.012906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.012879 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:13:27.096818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.096790 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx\" (UniqueName: \"kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx\") pod \"0ccb131f-d605-44f8-9f46-c86e995fce69\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " Oct 30 18:13:27.096992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.096875 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca\") pod \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " Oct 30 18:13:27.096992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.096902 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config\" (UniqueName: \"kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config\") pod \"10981b78-93cd-46b1-86ee-e648fd600bdb\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " Oct 30 18:13:27.096992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.096930 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca\") pod \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " Oct 30 18:13:27.096992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.096961 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984\") pod \"10981b78-93cd-46b1-86ee-e648fd600bdb\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " Oct 30 18:13:27.097209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.096991 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config\") pod \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " Oct 30 18:13:27.097209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.097042 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config\" (UniqueName: \"kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config\") pod \"0ccb131f-d605-44f8-9f46-c86e995fce69\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " Oct 30 18:13:27.097209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.097070 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk\" (UniqueName: \"kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk\") pod \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " Oct 30 18:13:27.097209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.097117 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls\" (UniqueName: \"kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls\") pod \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\" (UID: \"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6\") " Oct 30 18:13:27.097209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.097167 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls\" (UniqueName: \"kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls\") pod \"0ccb131f-d605-44f8-9f46-c86e995fce69\" (UID: \"0ccb131f-d605-44f8-9f46-c86e995fce69\") " Oct 30 18:13:27.097209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.097206 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls\" (UniqueName: \"kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls\") pod \"10981b78-93cd-46b1-86ee-e648fd600bdb\" (UID: \"10981b78-93cd-46b1-86ee-e648fd600bdb\") " Oct 30 18:13:27.098912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.098620 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config" pod "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" (UID: "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.098912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.098893 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca" pod "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" (UID: "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.099104 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.098994 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config" pod "10981b78-93cd-46b1-86ee-e648fd600bdb" (UID: "10981b78-93cd-46b1-86ee-e648fd600bdb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.099269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.099199 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca" pod "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" (UID: "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.099381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.099299 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config" pod "0ccb131f-d605-44f8-9f46-c86e995fce69" (UID: "0ccb131f-d605-44f8-9f46-c86e995fce69"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.101311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.101284 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls" pod "10981b78-93cd-46b1-86ee-e648fd600bdb" (UID: "10981b78-93cd-46b1-86ee-e648fd600bdb"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.101569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.101539 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls" pod "0ccb131f-d605-44f8-9f46-c86e995fce69" (UID: "0ccb131f-d605-44f8-9f46-c86e995fce69"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.101997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.101956 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984" pod "10981b78-93cd-46b1-86ee-e648fd600bdb" (UID: "10981b78-93cd-46b1-86ee-e648fd600bdb"). InnerVolumeSpecName "kube-api-access-6r984". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.103147 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.103085 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls" pod "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" (UID: "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.103562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.103537 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx" pod "0ccb131f-d605-44f8-9f46-c86e995fce69" (UID: "0ccb131f-d605-44f8-9f46-c86e995fce69"). InnerVolumeSpecName "kube-api-access-4lbhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.103662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.103625 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk" pod "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" (UID: "a097e0c7-aa30-4e0a-97ed-1c402da5a3d6"). InnerVolumeSpecName "kube-api-access-ggpmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.198236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198200 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0ccb131f-d605-44f8-9f46-c86e995fce69-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198236 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggpmk\" (UniqueName: \"kubernetes.io/projected/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-kube-api-access-ggpmk\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198252 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198262 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0ccb131f-d605-44f8-9f46-c86e995fce69-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198270 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/10981b78-93cd-46b1-86ee-e648fd600bdb-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198278 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lbhx\" (UniqueName: \"kubernetes.io/projected/0ccb131f-d605-44f8-9f46-c86e995fce69-kube-api-access-4lbhx\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198286 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-prometheus-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198294 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/10981b78-93cd-46b1-86ee-e648fd600bdb-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198302 6966 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-client-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198310 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6r984\" (UniqueName: \"kubernetes.io/projected/10981b78-93cd-46b1-86ee-e648fd600bdb-kube-api-access-6r984\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.198416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.198319 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.218781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.218720 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:13:27.219127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.219093 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="alertmanager" containerID="cri-o://a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0" gracePeriod=120 Oct 30 18:13:27.219650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.219153 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="config-reloader" containerID="cri-o://254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676" gracePeriod=120 Oct 30 18:13:27.239130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.238992 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:13:27.239318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.239283 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="prometheus" containerID="cri-o://feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" gracePeriod=600 Oct 30 18:13:27.239411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.239358 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="thanos-sidecar" containerID="cri-o://cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" gracePeriod=600 Oct 30 18:13:27.239480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.239380 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="config-reloader" containerID="cri-o://25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" gracePeriod=600 Oct 30 18:13:27.249064 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.248774 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.480787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.480765 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:27.506448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.506420 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508562 6966 generic.go:356] "Generic (PLEG): container finished" podID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerID="8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" exitCode=0 Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508587 6966 generic.go:356] "Generic (PLEG): container finished" podID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerID="6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" exitCode=0 Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508597 6966 generic.go:356] "Generic (PLEG): container finished" podID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerID="88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" exitCode=0 Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508621 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508636 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerDied","Data":"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616"} Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508700 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerDied","Data":"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5"} Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508724 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerDied","Data":"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8"} Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508754 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp" event={"ID":"a097e0c7-aa30-4e0a-97ed-1c402da5a3d6","Type":"ContainerDied","Data":"d1fd8b8d79ef3d823b6cacd0a49a8e617485c458c16787db90e36622dee906fe"} Oct 30 18:13:27.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.508776 6966 scope.go:122] "RemoveContainer" containerID="8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.510868 6966 generic.go:356] "Generic (PLEG): container finished" podID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerID="f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" exitCode=0 Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.510891 6966 generic.go:356] "Generic (PLEG): container finished" podID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerID="840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" exitCode=0 Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.510899 6966 generic.go:356] "Generic (PLEG): container finished" podID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerID="0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" exitCode=0 Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.510933 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.510971 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerDied","Data":"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6"} Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.511027 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerDied","Data":"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e"} Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.511065 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerDied","Data":"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8"} Oct 30 18:13:27.511161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.511100 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh" event={"ID":"0ccb131f-d605-44f8-9f46-c86e995fce69","Type":"ContainerDied","Data":"1ce185e825e612e3105b94bdd536f7c2e3f67f189b8f6ef3e3e19d3ffb749122"} Oct 30 18:13:27.514379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514356 6966 generic.go:356] "Generic (PLEG): container finished" podID="96695215-60d4-43b7-891a-130ee4cf7055" containerID="cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" exitCode=0 Oct 30 18:13:27.514379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514374 6966 generic.go:356] "Generic (PLEG): container finished" podID="96695215-60d4-43b7-891a-130ee4cf7055" containerID="25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" exitCode=0 Oct 30 18:13:27.514379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514381 6966 generic.go:356] "Generic (PLEG): container finished" podID="96695215-60d4-43b7-891a-130ee4cf7055" containerID="feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" exitCode=0 Oct 30 18:13:27.514621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514431 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:13:27.514621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514444 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerDied","Data":"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8"} Oct 30 18:13:27.514621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514518 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerDied","Data":"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155"} Oct 30 18:13:27.514621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514558 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerDied","Data":"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52"} Oct 30 18:13:27.514621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.514592 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"96695215-60d4-43b7-891a-130ee4cf7055","Type":"ContainerDied","Data":"0c9d4f8f4d7afd550b44947249c42a0e5900fa57a56e6e108803be2cb418e4ba"} Oct 30 18:13:27.515531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.515514 6966 generic.go:356] "Generic (PLEG): container finished" podID="10981b78-93cd-46b1-86ee-e648fd600bdb" containerID="ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8" exitCode=0 Oct 30 18:13:27.515584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.515577 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" event={"ID":"10981b78-93cd-46b1-86ee-e648fd600bdb","Type":"ContainerDied","Data":"ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8"} Oct 30 18:13:27.515627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.515604 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" event={"ID":"10981b78-93cd-46b1-86ee-e648fd600bdb","Type":"ContainerDied","Data":"f1b5be5dd403d842c07d176bcdd9319ee877379d595e6b19c66caad6b83cf316"} Oct 30 18:13:27.515670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.515659 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c" Oct 30 18:13:27.518513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.518489 6966 generic.go:356] "Generic (PLEG): container finished" podID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerID="254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676" exitCode=0 Oct 30 18:13:27.518609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.518522 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerDied","Data":"254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676"} Oct 30 18:13:27.519415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.519400 6966 scope.go:122] "RemoveContainer" containerID="6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" Oct 30 18:13:27.527241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.527224 6966 scope.go:122] "RemoveContainer" containerID="88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" Oct 30 18:13:27.536308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.536284 6966 scope.go:122] "RemoveContainer" containerID="8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" Oct 30 18:13:27.536662 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.536635 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": container with ID starting with 8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616 not found: ID does not exist" containerID="8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" Oct 30 18:13:27.536768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.536669 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616"} err="failed to get container status \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": rpc error: code = NotFound desc = could not find container \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": container with ID starting with 8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616 not found: ID does not exist" Oct 30 18:13:27.536768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.536687 6966 scope.go:122] "RemoveContainer" containerID="6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" Oct 30 18:13:27.536964 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.536945 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": container with ID starting with 6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5 not found: ID does not exist" containerID="6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" Oct 30 18:13:27.537024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.536969 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5"} err="failed to get container status \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": rpc error: code = NotFound desc = could not find container \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": container with ID starting with 6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5 not found: ID does not exist" Oct 30 18:13:27.537024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.536984 6966 scope.go:122] "RemoveContainer" containerID="88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" Oct 30 18:13:27.537273 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.537227 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": container with ID starting with 88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8 not found: ID does not exist" containerID="88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" Oct 30 18:13:27.537273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.537260 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8"} err="failed to get container status \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": rpc error: code = NotFound desc = could not find container \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": container with ID starting with 88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8 not found: ID does not exist" Oct 30 18:13:27.537388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.537280 6966 scope.go:122] "RemoveContainer" containerID="8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" Oct 30 18:13:27.537628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.537600 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616"} err="failed to get container status \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": rpc error: code = NotFound desc = could not find container \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": container with ID starting with 8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616 not found: ID does not exist" Oct 30 18:13:27.537712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.537629 6966 scope.go:122] "RemoveContainer" containerID="6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" Oct 30 18:13:27.537945 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.537921 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5"} err="failed to get container status \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": rpc error: code = NotFound desc = could not find container \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": container with ID starting with 6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5 not found: ID does not exist" Oct 30 18:13:27.538204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.537946 6966 scope.go:122] "RemoveContainer" containerID="88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" Oct 30 18:13:27.538528 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.538502 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8"} err="failed to get container status \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": rpc error: code = NotFound desc = could not find container \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": container with ID starting with 88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8 not found: ID does not exist" Oct 30 18:13:27.538621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.538527 6966 scope.go:122] "RemoveContainer" containerID="8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616" Oct 30 18:13:27.538818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.538786 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616"} err="failed to get container status \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": rpc error: code = NotFound desc = could not find container \"8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616\": container with ID starting with 8300e85dea0b2e8163d459cd326cc6c1d4670d43a25c68b854ee76c3f752f616 not found: ID does not exist" Oct 30 18:13:27.538818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.538807 6966 scope.go:122] "RemoveContainer" containerID="6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5" Oct 30 18:13:27.539099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.539076 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5"} err="failed to get container status \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": rpc error: code = NotFound desc = could not find container \"6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5\": container with ID starting with 6f879f437fb33b62ee11636ec54d845dc96a3e8489b88becbf87f4c834d322d5 not found: ID does not exist" Oct 30 18:13:27.539099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.539097 6966 scope.go:122] "RemoveContainer" containerID="88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8" Oct 30 18:13:27.539437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.539410 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8"} err="failed to get container status \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": rpc error: code = NotFound desc = could not find container \"88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8\": container with ID starting with 88ea47645879214c5ddaf55294e2d9ac3d1ac1e99f1d6a21561d4d4f857386e8 not found: ID does not exist" Oct 30 18:13:27.539437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.539435 6966 scope.go:122] "RemoveContainer" containerID="f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" Oct 30 18:13:27.549205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.549188 6966 scope.go:122] "RemoveContainer" containerID="840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" Oct 30 18:13:27.557436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.557417 6966 scope.go:122] "RemoveContainer" containerID="0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" Oct 30 18:13:27.564796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.564772 6966 scope.go:122] "RemoveContainer" containerID="f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" Oct 30 18:13:27.565078 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.565054 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": container with ID starting with f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6 not found: ID does not exist" containerID="f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" Oct 30 18:13:27.565132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565088 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6"} err="failed to get container status \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": rpc error: code = NotFound desc = could not find container \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": container with ID starting with f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6 not found: ID does not exist" Oct 30 18:13:27.565132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565110 6966 scope.go:122] "RemoveContainer" containerID="840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" Oct 30 18:13:27.565369 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.565350 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": container with ID starting with 840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e not found: ID does not exist" containerID="840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" Oct 30 18:13:27.565434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565379 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e"} err="failed to get container status \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": rpc error: code = NotFound desc = could not find container \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": container with ID starting with 840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e not found: ID does not exist" Oct 30 18:13:27.565434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565399 6966 scope.go:122] "RemoveContainer" containerID="0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" Oct 30 18:13:27.565662 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.565636 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": container with ID starting with 0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8 not found: ID does not exist" containerID="0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" Oct 30 18:13:27.565761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565668 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8"} err="failed to get container status \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": rpc error: code = NotFound desc = could not find container \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": container with ID starting with 0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8 not found: ID does not exist" Oct 30 18:13:27.565761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565688 6966 scope.go:122] "RemoveContainer" containerID="f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" Oct 30 18:13:27.565947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565928 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6"} err="failed to get container status \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": rpc error: code = NotFound desc = could not find container \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": container with ID starting with f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6 not found: ID does not exist" Oct 30 18:13:27.566018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.565949 6966 scope.go:122] "RemoveContainer" containerID="840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" Oct 30 18:13:27.566200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.566183 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e"} err="failed to get container status \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": rpc error: code = NotFound desc = could not find container \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": container with ID starting with 840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e not found: ID does not exist" Oct 30 18:13:27.566286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.566201 6966 scope.go:122] "RemoveContainer" containerID="0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" Oct 30 18:13:27.566471 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.566440 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8"} err="failed to get container status \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": rpc error: code = NotFound desc = could not find container \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": container with ID starting with 0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8 not found: ID does not exist" Oct 30 18:13:27.566538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.566485 6966 scope.go:122] "RemoveContainer" containerID="f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6" Oct 30 18:13:27.566779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.566750 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6"} err="failed to get container status \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": rpc error: code = NotFound desc = could not find container \"f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6\": container with ID starting with f035e09325aab58f3be49d58c10a00f50a3e15ceb87a168a905629c48bb669f6 not found: ID does not exist" Oct 30 18:13:27.566835 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.566782 6966 scope.go:122] "RemoveContainer" containerID="840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e" Oct 30 18:13:27.567072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.567052 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e"} err="failed to get container status \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": rpc error: code = NotFound desc = could not find container \"840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e\": container with ID starting with 840308a36083014f8bdbb7f5115495791d34034d51abdba018c91e12ba0d7f8e not found: ID does not exist" Oct 30 18:13:27.567156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.567073 6966 scope.go:122] "RemoveContainer" containerID="0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8" Oct 30 18:13:27.567326 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.567309 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8"} err="failed to get container status \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": rpc error: code = NotFound desc = could not find container \"0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8\": container with ID starting with 0688fef5bdf0122cdcb20d1916d81c722759c6f1bba62ddaaa0cfb2e83e000c8 not found: ID does not exist" Oct 30 18:13:27.567405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.567327 6966 scope.go:122] "RemoveContainer" containerID="cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" Oct 30 18:13:27.575367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.575353 6966 scope.go:122] "RemoveContainer" containerID="25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" Oct 30 18:13:27.579045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.579025 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp"] Oct 30 18:13:27.583556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.583537 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-mppnp"] Oct 30 18:13:27.584203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.584187 6966 scope.go:122] "RemoveContainer" containerID="feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" Oct 30 18:13:27.591899 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.591881 6966 scope.go:122] "RemoveContainer" containerID="a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5" Oct 30 18:13:27.599967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.599941 6966 scope.go:122] "RemoveContainer" containerID="cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" Oct 30 18:13:27.600246 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.600228 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": container with ID starting with cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8 not found: ID does not exist" containerID="cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" Oct 30 18:13:27.600313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.600251 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8"} err="failed to get container status \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": rpc error: code = NotFound desc = could not find container \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": container with ID starting with cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8 not found: ID does not exist" Oct 30 18:13:27.600313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.600266 6966 scope.go:122] "RemoveContainer" containerID="25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" Oct 30 18:13:27.600494 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.600474 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": container with ID starting with 25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155 not found: ID does not exist" containerID="25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" Oct 30 18:13:27.600551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.600496 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155"} err="failed to get container status \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": rpc error: code = NotFound desc = could not find container \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": container with ID starting with 25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155 not found: ID does not exist" Oct 30 18:13:27.600551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.600508 6966 scope.go:122] "RemoveContainer" containerID="feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" Oct 30 18:13:27.600739 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.600722 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": container with ID starting with feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52 not found: ID does not exist" containerID="feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" Oct 30 18:13:27.600783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.600743 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52"} err="failed to get container status \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": rpc error: code = NotFound desc = could not find container \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": container with ID starting with feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52 not found: ID does not exist" Oct 30 18:13:27.600783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.600757 6966 scope.go:122] "RemoveContainer" containerID="a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5" Oct 30 18:13:27.601034 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.601017 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": container with ID starting with a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5 not found: ID does not exist" containerID="a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5" Oct 30 18:13:27.601168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601037 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5"} err="failed to get container status \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": rpc error: code = NotFound desc = could not find container \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": container with ID starting with a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5 not found: ID does not exist" Oct 30 18:13:27.601168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601051 6966 scope.go:122] "RemoveContainer" containerID="cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" Oct 30 18:13:27.601168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601090 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601145 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601177 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601213 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601244 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601309 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601311 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8"} err="failed to get container status \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": rpc error: code = NotFound desc = could not find container \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": container with ID starting with cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8 not found: ID does not exist" Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601339 6966 scope.go:122] "RemoveContainer" containerID="25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" Oct 30 18:13:27.601381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601365 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601447 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601553 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601587 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out\" (UniqueName: \"kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601648 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601680 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.601779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601773 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"96695215-60d4-43b7-891a-130ee4cf7055\" (UID: \"96695215-60d4-43b7-891a-130ee4cf7055\") " Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602143 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155"} err="failed to get container status \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": rpc error: code = NotFound desc = could not find container \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": container with ID starting with 25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155 not found: ID does not exist" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602170 6966 scope.go:122] "RemoveContainer" containerID="feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.601473 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602571 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52"} err="failed to get container status \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": rpc error: code = NotFound desc = could not find container \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": container with ID starting with feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52 not found: ID does not exist" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602597 6966 scope.go:122] "RemoveContainer" containerID="a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602948 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5"} err="failed to get container status \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": rpc error: code = NotFound desc = could not find container \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": container with ID starting with a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5 not found: ID does not exist" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602955 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.602972 6966 scope.go:122] "RemoveContainer" containerID="cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.603256 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.603355 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8"} err="failed to get container status \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": rpc error: code = NotFound desc = could not find container \"cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8\": container with ID starting with cad9cc94428415dc4762eb08cbdf31524d5d7258b790647c5de689cdbaa2ccc8 not found: ID does not exist" Oct 30 18:13:27.603479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.603378 6966 scope.go:122] "RemoveContainer" containerID="25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155" Oct 30 18:13:27.604345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604169 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155"} err="failed to get container status \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": rpc error: code = NotFound desc = could not find container \"25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155\": container with ID starting with 25e1ee882012cf51863b8d67b3036e97407c95fb9f5f43e298d902d2947aa155 not found: ID does not exist" Oct 30 18:13:27.604345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604197 6966 scope.go:122] "RemoveContainer" containerID="feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52" Oct 30 18:13:27.604345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604312 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.604582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604539 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52"} err="failed to get container status \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": rpc error: code = NotFound desc = could not find container \"feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52\": container with ID starting with feb78271a1d6077e438414ac3d8fe51e665797c4a42fc85685dabf887c2ddc52 not found: ID does not exist" Oct 30 18:13:27.604582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604582 6966 scope.go:122] "RemoveContainer" containerID="a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5" Oct 30 18:13:27.604892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604865 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5"} err="failed to get container status \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": rpc error: code = NotFound desc = could not find container \"a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5\": container with ID starting with a8211e1dcd50f5699df1da7af539452df069ca0f7b2071d0bc66c59ba48737c5 not found: ID does not exist" Oct 30 18:13:27.604892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.604892 6966 scope.go:122] "RemoveContainer" containerID="ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8" Oct 30 18:13:27.605911 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.605885 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.606141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.606051 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.606141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.606077 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.606276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.606193 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.606601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.606451 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "kube-api-access-mxj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.606748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.606724 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:27.607585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.607563 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:27.607949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.607926 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:13:27.613019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.613003 6966 scope.go:122] "RemoveContainer" containerID="ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8" Oct 30 18:13:27.613360 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:27.613337 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8\": container with ID starting with ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8 not found: ID does not exist" containerID="ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8" Oct 30 18:13:27.613486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.613367 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8"} err="failed to get container status \"ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8\": rpc error: code = NotFound desc = could not find container \"ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8\": container with ID starting with ef5eaf81f06121244a4ecd45e084e89c9141b6a86fe7ab49a5482b9195c885d8 not found: ID does not exist" Oct 30 18:13:27.615408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.615386 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c"] Oct 30 18:13:27.617414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.617363 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config" pod "96695215-60d4-43b7-891a-130ee4cf7055" (UID: "96695215-60d4-43b7-891a-130ee4cf7055"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:27.619507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.619487 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-hck8c"] Oct 30 18:13:27.652199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.652176 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh"] Oct 30 18:13:27.657686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.657658 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-4j6nh"] Oct 30 18:13:27.703217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703198 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703218 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703229 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703238 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703246 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703255 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703264 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703274 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxj6h\" (UniqueName: \"kubernetes.io/projected/96695215-60d4-43b7-891a-130ee4cf7055-kube-api-access-mxj6h\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703281 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/96695215-60d4-43b7-891a-130ee4cf7055-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703290 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703299 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/96695215-60d4-43b7-891a-130ee4cf7055-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703323 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:13:27.703501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.703333 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/96695215-60d4-43b7-891a-130ee4cf7055-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:27.717874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.717861 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:13:27.737333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.737284 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:27.804492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.804447 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-83df9dfad9fd2ac35dca1441af096f35871cdbdee4927aba3c5e3091d60e99b5\"" Oct 30 18:13:27.861494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.861446 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:13:27.866821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.866800 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:13:27.889949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.889919 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:13:27.891177 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:27.891144 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="1f730bb1-7d33-4e72-a0f6-240280561cd5" containerName="otc-container" containerID="cri-o://6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6" gracePeriod=30 Oct 30 18:13:28.048194 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.048101 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:28.164243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.164215 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:13:28.167071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.167051 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:28.278514 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.278422 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:28.310344 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310316 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh\" (UniqueName: \"kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh\") pod \"1f730bb1-7d33-4e72-a0f6-240280561cd5\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " Oct 30 18:13:28.310480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310365 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs\") pod \"1f730bb1-7d33-4e72-a0f6-240280561cd5\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " Oct 30 18:13:28.310480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310420 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.310601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310493 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.310601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310523 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.310601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310550 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.310770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310609 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs\") pod \"1f730bb1-7d33-4e72-a0f6-240280561cd5\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " Oct 30 18:13:28.310770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310681 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.310770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310716 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal\") pod \"1f730bb1-7d33-4e72-a0f6-240280561cd5\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " Oct 30 18:13:28.310770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310744 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.310977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310785 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs\") pod \"1f730bb1-7d33-4e72-a0f6-240280561cd5\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " Oct 30 18:13:28.310977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310826 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert\") pod \"1f730bb1-7d33-4e72-a0f6-240280561cd5\" (UID: \"1f730bb1-7d33-4e72-a0f6-240280561cd5\") " Oct 30 18:13:28.310977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310825 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:28.310977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.310858 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config\") pod \"9aa3074f-51be-4249-80b7-255fbd4b3205\" (UID: \"9aa3074f-51be-4249-80b7-255fbd4b3205\") " Oct 30 18:13:28.311198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.311137 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.311774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.311603 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal" pod "1f730bb1-7d33-4e72-a0f6-240280561cd5" (UID: "1f730bb1-7d33-4e72-a0f6-240280561cd5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:28.313053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.313020 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh" pod "1f730bb1-7d33-4e72-a0f6-240280561cd5" (UID: "1f730bb1-7d33-4e72-a0f6-240280561cd5"). InnerVolumeSpecName "kube-api-access-x2hzh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.314199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.314170 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs" pod "1f730bb1-7d33-4e72-a0f6-240280561cd5" (UID: "1f730bb1-7d33-4e72-a0f6-240280561cd5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.314199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.314179 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.314554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.314528 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:13:28.314816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.314796 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs" pod "1f730bb1-7d33-4e72-a0f6-240280561cd5" (UID: "1f730bb1-7d33-4e72-a0f6-240280561cd5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.315302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.315269 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.315406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.315348 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs" pod "1f730bb1-7d33-4e72-a0f6-240280561cd5" (UID: "1f730bb1-7d33-4e72-a0f6-240280561cd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.315503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.315436 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "kube-api-access-dss9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.315751 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.315727 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert" pod "1f730bb1-7d33-4e72-a0f6-240280561cd5" (UID: "1f730bb1-7d33-4e72-a0f6-240280561cd5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.319703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.319682 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.327037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.327018 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config" pod "9aa3074f-51be-4249-80b7-255fbd4b3205" (UID: "9aa3074f-51be-4249-80b7-255fbd4b3205"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.411994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.411966 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.411994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.411991 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9aa3074f-51be-4249-80b7-255fbd4b3205-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412005 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/1f730bb1-7d33-4e72-a0f6-240280561cd5-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412019 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dss9x\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-kube-api-access-dss9x\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412031 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412046 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412059 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412073 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2hzh\" (UniqueName: \"kubernetes.io/projected/1f730bb1-7d33-4e72-a0f6-240280561cd5-kube-api-access-x2hzh\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412086 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/1f730bb1-7d33-4e72-a0f6-240280561cd5-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412098 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412110 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9aa3074f-51be-4249-80b7-255fbd4b3205-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.412121 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/9aa3074f-51be-4249-80b7-255fbd4b3205-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.527289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.527264 6966 generic.go:356] "Generic (PLEG): container finished" podID="1f730bb1-7d33-4e72-a0f6-240280561cd5" containerID="6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6" exitCode=0 Oct 30 18:13:28.527407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.527311 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:13:28.527407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.527340 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f730bb1-7d33-4e72-a0f6-240280561cd5","Type":"ContainerDied","Data":"6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6"} Oct 30 18:13:28.527407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.527383 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"1f730bb1-7d33-4e72-a0f6-240280561cd5","Type":"ContainerDied","Data":"5830527d160480c2d4bcb6b511bb87b32d95b4e3b834559033e24e51470c33e2"} Oct 30 18:13:28.527574 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.527410 6966 scope.go:122] "RemoveContainer" containerID="6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6" Oct 30 18:13:28.529594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.529530 6966 generic.go:356] "Generic (PLEG): container finished" podID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerID="a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0" exitCode=0 Oct 30 18:13:28.529702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.529588 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerDied","Data":"a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0"} Oct 30 18:13:28.529702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.529614 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:13:28.529702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.529647 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"9aa3074f-51be-4249-80b7-255fbd4b3205","Type":"ContainerDied","Data":"562f01412d7d0a9fc6d45ae163eae39451d3b44401111f251304e690552427e6"} Oct 30 18:13:28.538078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.538061 6966 scope.go:122] "RemoveContainer" containerID="6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6" Oct 30 18:13:28.538366 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:28.538346 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6\": container with ID starting with 6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6 not found: ID does not exist" containerID="6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6" Oct 30 18:13:28.538509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.538372 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6"} err="failed to get container status \"6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6\": rpc error: code = NotFound desc = could not find container \"6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6\": container with ID starting with 6d70e90693605132a2b02efee5c97004baba7a35350e7f701a5626176e10acc6 not found: ID does not exist" Oct 30 18:13:28.538509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.538393 6966 scope.go:122] "RemoveContainer" containerID="254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676" Oct 30 18:13:28.546052 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.546034 6966 scope.go:122] "RemoveContainer" containerID="a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0" Oct 30 18:13:28.553624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.553603 6966 scope.go:122] "RemoveContainer" containerID="66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf" Oct 30 18:13:28.560756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.560741 6966 scope.go:122] "RemoveContainer" containerID="254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676" Oct 30 18:13:28.561016 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:28.560999 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676\": container with ID starting with 254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676 not found: ID does not exist" containerID="254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676" Oct 30 18:13:28.561079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.561029 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676"} err="failed to get container status \"254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676\": rpc error: code = NotFound desc = could not find container \"254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676\": container with ID starting with 254e45c5aaa707a9ac93ce07a70e58b4d52081f497a9242cc658d928b8ad7676 not found: ID does not exist" Oct 30 18:13:28.561079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.561043 6966 scope.go:122] "RemoveContainer" containerID="a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0" Oct 30 18:13:28.561279 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:28.561264 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0\": container with ID starting with a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0 not found: ID does not exist" containerID="a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0" Oct 30 18:13:28.561320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.561282 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0"} err="failed to get container status \"a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0\": rpc error: code = NotFound desc = could not find container \"a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0\": container with ID starting with a2c1ba2fc76a1274f599ef13a068f6b306f98fb6eb760899c7b8285588dcbdc0 not found: ID does not exist" Oct 30 18:13:28.561320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.561296 6966 scope.go:122] "RemoveContainer" containerID="66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf" Oct 30 18:13:28.561557 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:28.561531 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf\": container with ID starting with 66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf not found: ID does not exist" containerID="66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf" Oct 30 18:13:28.561619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.561563 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf"} err="failed to get container status \"66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf\": rpc error: code = NotFound desc = could not find container \"66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf\": container with ID starting with 66f7edfad4fc491e260ddbb88cc9560e0ee90e79ab5ad2e62b4661fef82c4fbf not found: ID does not exist" Oct 30 18:13:28.574849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.574830 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:13:28.582023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.582003 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:13:28.619374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.619351 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:13:28.623502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.623484 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:13:28.908190 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:28.908150 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:29.128228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.128197 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:29.417411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.417370 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" path="/var/lib/kubelet/pods/0ccb131f-d605-44f8-9f46-c86e995fce69/volumes" Oct 30 18:13:29.418157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.418140 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10981b78-93cd-46b1-86ee-e648fd600bdb" path="/var/lib/kubelet/pods/10981b78-93cd-46b1-86ee-e648fd600bdb/volumes" Oct 30 18:13:29.418508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.418494 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f730bb1-7d33-4e72-a0f6-240280561cd5" path="/var/lib/kubelet/pods/1f730bb1-7d33-4e72-a0f6-240280561cd5/volumes" Oct 30 18:13:29.419220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.419195 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96695215-60d4-43b7-891a-130ee4cf7055" path="/var/lib/kubelet/pods/96695215-60d4-43b7-891a-130ee4cf7055/volumes" Oct 30 18:13:29.419696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.419681 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" path="/var/lib/kubelet/pods/9aa3074f-51be-4249-80b7-255fbd4b3205/volumes" Oct 30 18:13:29.420354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:29.420340 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" path="/var/lib/kubelet/pods/a097e0c7-aa30-4e0a-97ed-1c402da5a3d6/volumes" Oct 30 18:13:31.869790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.869761 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-8zxxj"] Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870088 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="kube-rbac-proxy" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870099 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="kube-rbac-proxy" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870109 6966 cpu_manager.go:414] "containerMap: removing container" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="thanos-sidecar" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870114 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="thanos-sidecar" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870128 6966 cpu_manager.go:414] "containerMap: removing container" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="init-config-reloader" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870133 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="init-config-reloader" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870142 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="alertmanager" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870147 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="alertmanager" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870153 6966 cpu_manager.go:414] "containerMap: removing container" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="prometheus" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870159 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="prometheus" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870165 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="prom-label-proxy" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870171 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="prom-label-proxy" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870182 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="method-gate" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870187 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="method-gate" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870195 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="prom-label-proxy" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870199 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="prom-label-proxy" Oct 30 18:13:31.870200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870207 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1f730bb1-7d33-4e72-a0f6-240280561cd5" containerName="otc-container" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870212 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f730bb1-7d33-4e72-a0f6-240280561cd5" containerName="otc-container" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870218 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870222 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870232 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="kube-rbac-proxy" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870236 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="kube-rbac-proxy" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870246 6966 cpu_manager.go:414] "containerMap: removing container" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870250 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870255 6966 cpu_manager.go:414] "containerMap: removing container" podUID="10981b78-93cd-46b1-86ee-e648fd600bdb" containerName="korrel8r" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870259 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10981b78-93cd-46b1-86ee-e648fd600bdb" containerName="korrel8r" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870266 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="method-gate" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870270 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="method-gate" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870276 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="init-config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870280 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="init-config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870329 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="alertmanager" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870337 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="prometheus" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870342 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="prom-label-proxy" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870349 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="method-gate" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870355 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="prom-label-proxy" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870361 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f730bb1-7d33-4e72-a0f6-240280561cd5" containerName="otc-container" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870366 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9aa3074f-51be-4249-80b7-255fbd4b3205" containerName="config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870372 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="kube-rbac-proxy" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870377 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="config-reloader" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870382 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="10981b78-93cd-46b1-86ee-e648fd600bdb" containerName="korrel8r" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870386 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a097e0c7-aa30-4e0a-97ed-1c402da5a3d6" containerName="method-gate" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870393 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ccb131f-d605-44f8-9f46-c86e995fce69" containerName="kube-rbac-proxy" Oct 30 18:13:31.870900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.870399 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="96695215-60d4-43b7-891a-130ee4cf7055" containerName="thanos-sidecar" Oct 30 18:13:31.875299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.875283 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:31.893155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.893134 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-8zxxj"] Oct 30 18:13:31.903055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.902991 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:13:31.903055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.903016 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:13:31.903055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.902991 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:13:31.903239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.903062 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:13:31.903239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.903143 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-cmgfj\"" Oct 30 18:13:31.921836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:31.921810 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:32.042260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.042237 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd5x8\" (UniqueName: \"kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.042355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.042267 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.042355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.042298 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.143232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.143177 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd5x8\" (UniqueName: \"kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.143232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.143217 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.143355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.143253 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.143989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.143969 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.145642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.145622 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.146648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.146623 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:32.165502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.165477 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd5x8\" (UniqueName: \"kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8\") pod \"korrel8r-6644769df5-8zxxj\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.187818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.187795 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.486945 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.486862 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-8zxxj"] Oct 30 18:13:32.492761 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:13:32.492729 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2c3114_d07f_4473_a475_dc8072c2abfc.slice/crio-ca4e002dd233fe32b3f032314b4051942da15ee95468ab45e5c15df290785c3e WatchSource:0}: Error finding container ca4e002dd233fe32b3f032314b4051942da15ee95468ab45e5c15df290785c3e: Status 404 returned error can't find the container with id ca4e002dd233fe32b3f032314b4051942da15ee95468ab45e5c15df290785c3e Oct 30 18:13:32.497645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.497622 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:13:32.502719 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.502697 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.554025 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.553999 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" event={"ID":"1e2c3114-d07f-4473-a475-dc8072c2abfc","Type":"ContainerStarted","Data":"bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084"} Oct 30 18:13:32.554111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.554050 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" event={"ID":"1e2c3114-d07f-4473-a475-dc8072c2abfc","Type":"ContainerStarted","Data":"ca4e002dd233fe32b3f032314b4051942da15ee95468ab45e5c15df290785c3e"} Oct 30 18:13:32.554111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.554081 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:32.592443 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.592421 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:13:32.592686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.592670 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:13:32.603391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.603369 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-zr47k\"" Oct 30 18:13:32.603512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.603485 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:13:32.605051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.605031 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:13:32.647439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.647412 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.647578 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.647545 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc4gb\" (UniqueName: \"kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.647689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.647668 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.647764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.647720 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.664720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.664687 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:32.671287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.671252 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" podStartSLOduration=1.671241451 podStartE2EDuration="1.671241451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:32.670787236 +0000 UTC m=+975.376972594" watchObservedRunningTime="2026-10-30 18:13:32.671241451 +0000 UTC m=+975.377426822" Oct 30 18:13:32.748840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.748769 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc4gb\" (UniqueName: \"kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.748840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.748824 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.749014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.748874 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.749014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.748907 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.749598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.749579 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.751326 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.751308 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.751384 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.751335 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.771509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.771491 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc4gb\" (UniqueName: \"kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb\") pod \"data-science-collector-collector-0\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.828582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.828561 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:13:32.878948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:32.878916 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:33.138339 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.138306 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:13:33.143174 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:13:33.143149 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e57cfcb_1c7d_451f_b3ad_991d863bc56c.slice/crio-e8287dca47c668816f94f86be60a55d0046d6f44ce604476fbd7971e92b92a4c WatchSource:0}: Error finding container e8287dca47c668816f94f86be60a55d0046d6f44ce604476fbd7971e92b92a4c: Status 404 returned error can't find the container with id e8287dca47c668816f94f86be60a55d0046d6f44ce604476fbd7971e92b92a4c Oct 30 18:13:33.145205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.145174 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:13:33.145290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.145243 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:13:33.566384 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.562667 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9e57cfcb-1c7d-451f-b3ad-991d863bc56c","Type":"ContainerStarted","Data":"728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e"} Oct 30 18:13:33.566384 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.562734 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9e57cfcb-1c7d-451f-b3ad-991d863bc56c","Type":"ContainerStarted","Data":"e8287dca47c668816f94f86be60a55d0046d6f44ce604476fbd7971e92b92a4c"} Oct 30 18:13:33.604329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.604279 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.6042668789999999 podStartE2EDuration="1.604266879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:33.603775553 +0000 UTC m=+976.309960912" watchObservedRunningTime="2026-10-30 18:13:33.604266879 +0000 UTC m=+976.310452235" Oct 30 18:13:33.617211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.617184 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:13:33.621351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.621331 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.637917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.637881 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:13:33.638030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.637958 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:13:33.638084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.638037 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:13:33.638167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.638146 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:13:33.638282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.638191 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:13:33.638282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.638200 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:13:33.638282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.638215 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:13:33.638396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.638304 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-845zf\"" Oct 30 18:13:33.646766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.646740 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:13:33.756015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.755976 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:33.757060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757035 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757073 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757186 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757220 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757333 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757410 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757445 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.757525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.757498 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crr5c\" (UniqueName: \"kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.858866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.858794 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.858866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.858840 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.858938 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.858963 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crr5c\" (UniqueName: \"kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.858986 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859017 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859266 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859161 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859266 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859204 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859852 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.859890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859860 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.860061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859904 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.860061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.859928 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.861451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.861425 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.861552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.861489 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.861589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.861565 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:33.880649 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:33.880631 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crr5c\" (UniqueName: \"kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:34.015013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:34.014977 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:13:36.277337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.277305 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.305121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.305086 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.378521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.378492 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.378654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.378591 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.392343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.392320 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") DevicePath \"csi-f1d243cd1eff0987a64de94f5fb75fb003a2545b22342d982b5bcfa8f4a85c68\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.494158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.494128 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.520619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.520599 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.641685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.641661 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:36.972352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:36.972241 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:13:36.976949 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:13:36.976917 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a8755b5_ff7b_466f_8c6b_ff2a478e0789.slice/crio-4e198055464d277b664ede91cdf9d3183a0947a29032324ff2d49610fadc68b2 WatchSource:0}: Error finding container 4e198055464d277b664ede91cdf9d3183a0947a29032324ff2d49610fadc68b2: Status 404 returned error can't find the container with id 4e198055464d277b664ede91cdf9d3183a0947a29032324ff2d49610fadc68b2 Oct 30 18:13:37.582309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.582279 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerStarted","Data":"e37250ef247f900eb2920dbd93ccd711fbb73757f7772e159907a039a464037b"} Oct 30 18:13:37.582810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.582336 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerStarted","Data":"a12e7efc1510021ae1daf962043bab13fa6846f5cac275981d8548e41afd22d9"} Oct 30 18:13:37.582810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.582362 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerStarted","Data":"c3c37616425f341663ed0a1d6830f5fc0d10242d26e73936704ce890ba3a9755"} Oct 30 18:13:37.582810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.582392 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:37.582810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.582417 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerStarted","Data":"4e198055464d277b664ede91cdf9d3183a0947a29032324ff2d49610fadc68b2"} Oct 30 18:13:37.584865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.584842 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:37.650120 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:37.650067 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=4.650055555 podStartE2EDuration="4.650055555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:37.637790375 +0000 UTC m=+980.343975752" watchObservedRunningTime="2026-10-30 18:13:37.650055555 +0000 UTC m=+980.356240912" Oct 30 18:13:38.587765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:38.587734 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:38.588165 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:38.587813 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:38.596238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:38.596216 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:13:43.564001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:13:43.563962 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:13:47.199190 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:13:47.199148 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:14:00.601298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:00.601196 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:14:06.462585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:06.462543 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:08.126729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:08.126696 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:08.359389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:08.359352 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:10.605708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:10.605666 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:14:17.198503 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:14:17.198443 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:14:18.533826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:18.533791 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:20.601416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:20.601388 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:21.821365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:21.821336 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-8zxxj"] Oct 30 18:14:21.821815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:21.821574 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" podUID="1e2c3114-d07f-4473-a475-dc8072c2abfc" containerName="korrel8r" containerID="cri-o://bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084" gracePeriod=30 Oct 30 18:14:22.074051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.074000 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:14:22.222887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.222839 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:14:22.223276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.223082 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="9e57cfcb-1c7d-451f-b3ad-991d863bc56c" containerName="otc-container" containerID="cri-o://728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e" gracePeriod=30 Oct 30 18:14:22.230510 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.230480 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8\" (UniqueName: \"kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8\") pod \"1e2c3114-d07f-4473-a475-dc8072c2abfc\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " Oct 30 18:14:22.230716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.230575 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config\" (UniqueName: \"kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config\") pod \"1e2c3114-d07f-4473-a475-dc8072c2abfc\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " Oct 30 18:14:22.230716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.230616 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls\" (UniqueName: \"kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls\") pod \"1e2c3114-d07f-4473-a475-dc8072c2abfc\" (UID: \"1e2c3114-d07f-4473-a475-dc8072c2abfc\") " Oct 30 18:14:22.231557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.231520 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config" pod "1e2c3114-d07f-4473-a475-dc8072c2abfc" (UID: "1e2c3114-d07f-4473-a475-dc8072c2abfc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:22.235114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.235088 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls" pod "1e2c3114-d07f-4473-a475-dc8072c2abfc" (UID: "1e2c3114-d07f-4473-a475-dc8072c2abfc"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:22.249259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.249233 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8" pod "1e2c3114-d07f-4473-a475-dc8072c2abfc" (UID: "1e2c3114-d07f-4473-a475-dc8072c2abfc"). InnerVolumeSpecName "kube-api-access-bd5x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:22.331598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.331541 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1e2c3114-d07f-4473-a475-dc8072c2abfc-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.331598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.331572 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1e2c3114-d07f-4473-a475-dc8072c2abfc-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.331598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.331586 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd5x8\" (UniqueName: \"kubernetes.io/projected/1e2c3114-d07f-4473-a475-dc8072c2abfc-kube-api-access-bd5x8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.406322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.406291 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:22.462004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.461980 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:22.624322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.624237 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:22.634178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.634156 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs\") pod \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " Oct 30 18:14:22.634315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.634188 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs\") pod \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " Oct 30 18:14:22.634315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.634215 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal\") pod \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " Oct 30 18:14:22.634425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.634355 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb\" (UniqueName: \"kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb\") pod \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\" (UID: \"9e57cfcb-1c7d-451f-b3ad-991d863bc56c\") " Oct 30 18:14:22.634665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.634639 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal" pod "9e57cfcb-1c7d-451f-b3ad-991d863bc56c" (UID: "9e57cfcb-1c7d-451f-b3ad-991d863bc56c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:22.634769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.634753 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.636253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.636222 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs" pod "9e57cfcb-1c7d-451f-b3ad-991d863bc56c" (UID: "9e57cfcb-1c7d-451f-b3ad-991d863bc56c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:22.636334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.636290 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs" pod "9e57cfcb-1c7d-451f-b3ad-991d863bc56c" (UID: "9e57cfcb-1c7d-451f-b3ad-991d863bc56c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:22.636334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.636311 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb" pod "9e57cfcb-1c7d-451f-b3ad-991d863bc56c" (UID: "9e57cfcb-1c7d-451f-b3ad-991d863bc56c"). InnerVolumeSpecName "kube-api-access-zc4gb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:22.676422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.675516 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:14:22.676422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.675750 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo" containerID="cri-o://c3c37616425f341663ed0a1d6830f5fc0d10242d26e73936704ce890ba3a9755" gracePeriod=30 Oct 30 18:14:22.676422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.676088 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway-opa" containerID="cri-o://e37250ef247f900eb2920dbd93ccd711fbb73757f7772e159907a039a464037b" gracePeriod=30 Oct 30 18:14:22.676422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.676181 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway" containerID="cri-o://a12e7efc1510021ae1daf962043bab13fa6846f5cac275981d8548e41afd22d9" gracePeriod=30 Oct 30 18:14:22.736110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.736040 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc4gb\" (UniqueName: \"kubernetes.io/projected/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-kube-api-access-zc4gb\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.736110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.736071 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.736110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.736087 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e57cfcb-1c7d-451f-b3ad-991d863bc56c-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:22.813182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.813150 6966 generic.go:356] "Generic (PLEG): container finished" podID="1e2c3114-d07f-4473-a475-dc8072c2abfc" containerID="bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084" exitCode=0 Oct 30 18:14:22.813350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.813216 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" Oct 30 18:14:22.813350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.813232 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" event={"ID":"1e2c3114-d07f-4473-a475-dc8072c2abfc","Type":"ContainerDied","Data":"bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084"} Oct 30 18:14:22.813350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.813296 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-8zxxj" event={"ID":"1e2c3114-d07f-4473-a475-dc8072c2abfc","Type":"ContainerDied","Data":"ca4e002dd233fe32b3f032314b4051942da15ee95468ab45e5c15df290785c3e"} Oct 30 18:14:22.813350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.813324 6966 scope.go:122] "RemoveContainer" containerID="bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084" Oct 30 18:14:22.820529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.820508 6966 generic.go:356] "Generic (PLEG): container finished" podID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerID="e37250ef247f900eb2920dbd93ccd711fbb73757f7772e159907a039a464037b" exitCode=0 Oct 30 18:14:22.820634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.820582 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerDied","Data":"e37250ef247f900eb2920dbd93ccd711fbb73757f7772e159907a039a464037b"} Oct 30 18:14:22.822066 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.822045 6966 generic.go:356] "Generic (PLEG): container finished" podID="9e57cfcb-1c7d-451f-b3ad-991d863bc56c" containerID="728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e" exitCode=0 Oct 30 18:14:22.822422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.822083 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:14:22.822422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.822093 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9e57cfcb-1c7d-451f-b3ad-991d863bc56c","Type":"ContainerDied","Data":"728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e"} Oct 30 18:14:22.822422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.822131 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"9e57cfcb-1c7d-451f-b3ad-991d863bc56c","Type":"ContainerDied","Data":"e8287dca47c668816f94f86be60a55d0046d6f44ce604476fbd7971e92b92a4c"} Oct 30 18:14:22.823345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.823309 6966 scope.go:122] "RemoveContainer" containerID="bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084" Oct 30 18:14:22.823612 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:14:22.823591 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084\": container with ID starting with bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084 not found: ID does not exist" containerID="bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084" Oct 30 18:14:22.823694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.823624 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084"} err="failed to get container status \"bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084\": rpc error: code = NotFound desc = could not find container \"bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084\": container with ID starting with bd47e208f5b9a909bf6d90cd96bec0181ba0a22a895f3703d8659ee047b09084 not found: ID does not exist" Oct 30 18:14:22.823694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.823647 6966 scope.go:122] "RemoveContainer" containerID="728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e" Oct 30 18:14:22.830931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.830915 6966 scope.go:122] "RemoveContainer" containerID="728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e" Oct 30 18:14:22.831185 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:14:22.831167 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e\": container with ID starting with 728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e not found: ID does not exist" containerID="728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e" Oct 30 18:14:22.831246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.831189 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e"} err="failed to get container status \"728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e\": rpc error: code = NotFound desc = could not find container \"728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e\": container with ID starting with 728415429e100f58a382c891bfbaba67590acbba6c8639a50c649298165e4a7e not found: ID does not exist" Oct 30 18:14:22.844393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.844369 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:22.862506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.862486 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-8zxxj"] Oct 30 18:14:22.866260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.866241 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-8zxxj"] Oct 30 18:14:22.913117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.913098 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:14:22.921492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:22.921475 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:14:23.074301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.074263 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:23.410798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.410766 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2c3114-d07f-4473-a475-dc8072c2abfc" path="/var/lib/kubelet/pods/1e2c3114-d07f-4473-a475-dc8072c2abfc/volumes" Oct 30 18:14:23.411217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.411197 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e57cfcb-1c7d-451f-b3ad-991d863bc56c" path="/var/lib/kubelet/pods/9e57cfcb-1c7d-451f-b3ad-991d863bc56c/volumes" Oct 30 18:14:23.592534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.592506 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:14:23.829328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.829295 6966 generic.go:356] "Generic (PLEG): container finished" podID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerID="a12e7efc1510021ae1daf962043bab13fa6846f5cac275981d8548e41afd22d9" exitCode=0 Oct 30 18:14:23.829328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.829319 6966 generic.go:356] "Generic (PLEG): container finished" podID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerID="c3c37616425f341663ed0a1d6830f5fc0d10242d26e73936704ce890ba3a9755" exitCode=0 Oct 30 18:14:23.829889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.829371 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerDied","Data":"a12e7efc1510021ae1daf962043bab13fa6846f5cac275981d8548e41afd22d9"} Oct 30 18:14:23.829889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.829428 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerDied","Data":"c3c37616425f341663ed0a1d6830f5fc0d10242d26e73936704ce890ba3a9755"} Oct 30 18:14:23.929261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:23.929239 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:24.048758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.048680 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.048758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.048723 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.048982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.048763 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.048982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.048835 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.048982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.048869 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.048982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.048907 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c\" (UniqueName: \"kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.049196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049067 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:24.049309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049213 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.049413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049355 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.050235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049487 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\" (UID: \"1a8755b5-ff7b-466f-8c6b-ff2a478e0789\") " Oct 30 18:14:24.050235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049811 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:24.050235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049946 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.050235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.049969 6966 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.050235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.050085 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:24.050235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.050147 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:24.051499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.051448 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "kube-api-access-crr5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:24.051600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.051452 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:24.051600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.051514 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:24.051600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.051536 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:24.052708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.052682 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "1a8755b5-ff7b-466f-8c6b-ff2a478e0789" (UID: "1a8755b5-ff7b-466f-8c6b-ff2a478e0789"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:14:24.150989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.150968 6966 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-gateway-tenants\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.150989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.150989 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.151117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.151003 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crr5c\" (UniqueName: \"kubernetes.io/projected/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-kube-api-access-crr5c\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.151117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.151038 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:14:24.151117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.151051 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.151117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.151062 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.151117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.151072 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/1a8755b5-ff7b-466f-8c6b-ff2a478e0789-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:14:24.159882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.159866 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:14:24.251905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.251879 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-f1d243cd1eff0987a64de94f5fb75fb003a2545b22342d982b5bcfa8f4a85c68\"" Oct 30 18:14:24.838616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.838583 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"1a8755b5-ff7b-466f-8c6b-ff2a478e0789","Type":"ContainerDied","Data":"4e198055464d277b664ede91cdf9d3183a0947a29032324ff2d49610fadc68b2"} Oct 30 18:14:24.839093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.838622 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:14:24.839093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.838643 6966 scope.go:122] "RemoveContainer" containerID="e37250ef247f900eb2920dbd93ccd711fbb73757f7772e159907a039a464037b" Oct 30 18:14:24.849583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.849564 6966 scope.go:122] "RemoveContainer" containerID="a12e7efc1510021ae1daf962043bab13fa6846f5cac275981d8548e41afd22d9" Oct 30 18:14:24.860104 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.860089 6966 scope.go:122] "RemoveContainer" containerID="c3c37616425f341663ed0a1d6830f5fc0d10242d26e73936704ce890ba3a9755" Oct 30 18:14:24.899710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.899689 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:14:24.911179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:24.911159 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:14:25.410686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:25.410661 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" path="/var/lib/kubelet/pods/1a8755b5-ff7b-466f-8c6b-ff2a478e0789/volumes" Oct 30 18:14:31.069524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.069481 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:31.201218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201193 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c"] Oct 30 18:14:31.201555 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201535 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9e57cfcb-1c7d-451f-b3ad-991d863bc56c" containerName="otc-container" Oct 30 18:14:31.201617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201556 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e57cfcb-1c7d-451f-b3ad-991d863bc56c" containerName="otc-container" Oct 30 18:14:31.201617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201573 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway" Oct 30 18:14:31.201617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201580 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway" Oct 30 18:14:31.201617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201592 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1e2c3114-d07f-4473-a475-dc8072c2abfc" containerName="korrel8r" Oct 30 18:14:31.201617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201597 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2c3114-d07f-4473-a475-dc8072c2abfc" containerName="korrel8r" Oct 30 18:14:31.201617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201616 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201621 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201628 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway-opa" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201633 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway-opa" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201682 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2c3114-d07f-4473-a475-dc8072c2abfc" containerName="korrel8r" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201692 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201698 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway-opa" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201705 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e57cfcb-1c7d-451f-b3ad-991d863bc56c" containerName="otc-container" Oct 30 18:14:31.201796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.201711 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8755b5-ff7b-466f-8c6b-ff2a478e0789" containerName="tempo-gateway" Oct 30 18:14:31.207117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.207097 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.226237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.226215 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c"] Oct 30 18:14:31.243265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.243243 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg"] Oct 30 18:14:31.247155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.247138 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.267135 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.267115 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg"] Oct 30 18:14:31.280278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.280248 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:14:31.280278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.280260 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-nqf2m\"" Oct 30 18:14:31.280431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.280260 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-kv2jn\"" Oct 30 18:14:31.280431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.280319 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:14:31.284483 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.284450 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm"] Oct 30 18:14:31.287688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.287670 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.288846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.288825 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:31.292538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.292518 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:14:31.292653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.292559 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:14:31.292653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.292565 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:14:31.292653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.292519 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:14:31.292653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.292602 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:14:31.292653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.292607 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:14:31.302033 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.302015 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-v5vs9\"" Oct 30 18:14:31.302162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.302067 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:14:31.307701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.307680 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.307778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.307700 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm"] Oct 30 18:14:31.307778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.307711 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.307778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.307732 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.307885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.307859 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzzt4\" (UniqueName: \"kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.307919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.307886 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.392497 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.392422 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:14:31.408645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408620 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.408749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408666 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzzt4\" (UniqueName: \"kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.408749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408690 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.408831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408758 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggvxs\" (UniqueName: \"kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.408864 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408826 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.408902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408872 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.408969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.408951 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzwc\" (UniqueName: \"kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.409033 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409017 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.409101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409082 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.409169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409155 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.409215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409191 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.409560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409533 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.409648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409628 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.409856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.409837 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.411689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.411664 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.430979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.430957 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzzt4\" (UniqueName: \"kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.510308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510284 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggvxs\" (UniqueName: \"kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.510389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510329 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.510389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510363 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.510515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510410 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzwc\" (UniqueName: \"kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.510515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510484 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.510631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510518 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.511006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.510986 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.511306 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.511287 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.512563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.512538 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.512726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.512711 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.524700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.524678 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:14:31.532381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.532351 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzwc\" (UniqueName: \"kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc\") pod \"data-science-thanos-querier-proxy-6998ff8b98-h6cxg\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.532683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.532663 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggvxs\" (UniqueName: \"kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs\") pod \"korrel8r-66d6f7f696-mf4qm\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.563644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.563607 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:14:31.600382 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.600350 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:31.642739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.642672 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:31.834061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.834034 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c"] Oct 30 18:14:31.838381 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:14:31.838355 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod153c539e_4f11_438b_8175_cedabcc2c225.slice/crio-6c47e8ba6a1ca58b0c52d1d00b9f4bd915876b7564a2027d15ee971ac78ede72 WatchSource:0}: Error finding container 6c47e8ba6a1ca58b0c52d1d00b9f4bd915876b7564a2027d15ee971ac78ede72: Status 404 returned error can't find the container with id 6c47e8ba6a1ca58b0c52d1d00b9f4bd915876b7564a2027d15ee971ac78ede72 Oct 30 18:14:31.875529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.875500 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg"] Oct 30 18:14:31.878336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.878296 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerStarted","Data":"6c47e8ba6a1ca58b0c52d1d00b9f4bd915876b7564a2027d15ee971ac78ede72"} Oct 30 18:14:31.885749 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:14:31.885726 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dd5a275_5cc7_494f_a922_cddb8f6ff89f.slice/crio-6112b28aff7b26e3502a8f7bef299b9fc58718c66711483b1955362ad6842af6 WatchSource:0}: Error finding container 6112b28aff7b26e3502a8f7bef299b9fc58718c66711483b1955362ad6842af6: Status 404 returned error can't find the container with id 6112b28aff7b26e3502a8f7bef299b9fc58718c66711483b1955362ad6842af6 Oct 30 18:14:31.895808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.895782 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:31.904366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:31.904019 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm"] Oct 30 18:14:31.912556 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:14:31.912527 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96fd765e_ef8f_4dda_b423_fde2f39c0220.slice/crio-c03f6c11a8a6f4873c92b26409c24fd96fe843f27a17030f4fdb5d972f123c17 WatchSource:0}: Error finding container c03f6c11a8a6f4873c92b26409c24fd96fe843f27a17030f4fdb5d972f123c17: Status 404 returned error can't find the container with id c03f6c11a8a6f4873c92b26409c24fd96fe843f27a17030f4fdb5d972f123c17 Oct 30 18:14:32.113861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.113839 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:32.317729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.317694 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:32.846523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.846491 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:14:32.850446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.850420 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:32.871627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.871606 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:14:32.880432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.880401 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:14:32.880571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.880524 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:14:32.881196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.881176 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-mbgzb\"" Oct 30 18:14:32.881378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.881355 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:14:32.881378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.881354 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:14:32.888396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.888372 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerStarted","Data":"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d"} Oct 30 18:14:32.888523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.888428 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerStarted","Data":"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df"} Oct 30 18:14:32.888523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.888481 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerStarted","Data":"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af"} Oct 30 18:14:32.890085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.890059 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" event={"ID":"96fd765e-ef8f-4dda-b423-fde2f39c0220","Type":"ContainerStarted","Data":"1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3"} Oct 30 18:14:32.890175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.890105 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" event={"ID":"96fd765e-ef8f-4dda-b423-fde2f39c0220","Type":"ContainerStarted","Data":"c03f6c11a8a6f4873c92b26409c24fd96fe843f27a17030f4fdb5d972f123c17"} Oct 30 18:14:32.890175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.890150 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:32.892450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.892426 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerStarted","Data":"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2"} Oct 30 18:14:32.892590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.892505 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerStarted","Data":"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5"} Oct 30 18:14:32.892590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.892542 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerStarted","Data":"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a"} Oct 30 18:14:32.892590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.892577 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerStarted","Data":"6112b28aff7b26e3502a8f7bef299b9fc58718c66711483b1955362ad6842af6"} Oct 30 18:14:32.934585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.934546 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:32.937739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.937689 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" podStartSLOduration=1.937673489 podStartE2EDuration="1.937673489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:32.936241188 +0000 UTC m=+1035.642426545" watchObservedRunningTime="2026-10-30 18:14:32.937673489 +0000 UTC m=+1035.643858849" Oct 30 18:14:32.940049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.940031 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:14:32.977995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:32.977949 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" podStartSLOduration=1.977936906 podStartE2EDuration="1.977936906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:32.976377793 +0000 UTC m=+1035.682563152" watchObservedRunningTime="2026-10-30 18:14:32.977936906 +0000 UTC m=+1035.684122300" Oct 30 18:14:33.017947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.017897 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" podStartSLOduration=2.01788356 podStartE2EDuration="2.01788356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:33.016174302 +0000 UTC m=+1035.722359659" watchObservedRunningTime="2026-10-30 18:14:33.01788356 +0000 UTC m=+1035.724068916" Oct 30 18:14:33.022606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.022581 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.022729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.022631 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.022729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.022648 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xk5\" (UniqueName: \"kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.022819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.022796 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.022863 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.022836 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.023265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.023243 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124013 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124067 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124103 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124127 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xk5\" (UniqueName: \"kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124170 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124197 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.124927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.124896 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.126804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.126778 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.126977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.126962 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.127115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.127092 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.127157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.127100 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.145797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.145773 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xk5\" (UniqueName: \"kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5\") pod \"data-science-collector-collector-1\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.148012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.147984 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:33.165802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.165782 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:14:33.463077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.463015 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:14:33.468055 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:14:33.468033 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod125e9d51_9af0_4237_916d_8cba28e3f7a0.slice/crio-f4f3587a89b188c3790c1de7016b54272958664a4127b7b8ffa5a536c4e0156d WatchSource:0}: Error finding container f4f3587a89b188c3790c1de7016b54272958664a4127b7b8ffa5a536c4e0156d: Status 404 returned error can't find the container with id f4f3587a89b188c3790c1de7016b54272958664a4127b7b8ffa5a536c4e0156d Oct 30 18:14:33.470348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.470314 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:14:33.470401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.470376 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:14:33.899543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.899508 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"125e9d51-9af0-4237-916d-8cba28e3f7a0","Type":"ContainerStarted","Data":"78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417"} Oct 30 18:14:33.899736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.899562 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"125e9d51-9af0-4237-916d-8cba28e3f7a0","Type":"ContainerStarted","Data":"f4f3587a89b188c3790c1de7016b54272958664a4127b7b8ffa5a536c4e0156d"} Oct 30 18:14:33.945737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:33.945661 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.94564359 podStartE2EDuration="1.94564359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:33.94390941 +0000 UTC m=+1036.650094770" watchObservedRunningTime="2026-10-30 18:14:33.94564359 +0000 UTC m=+1036.651828948" Oct 30 18:14:34.385350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:34.385310 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:34.586952 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:34.586915 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.411603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.411566 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.695102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.695021 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.898656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.898625 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:40.972368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.972283 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:14:40.977306 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.977282 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:40.981517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.981487 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:14:40.985520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.985501 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:40.991303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.991283 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:14:40.991303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.991293 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:14:40.993955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:40.993940 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:14:41.006148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.006128 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:14:41.008661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.008630 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:14:41.008768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.008686 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:14:41.008873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.008855 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:14:41.015710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.015686 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:14:41.086297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086267 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.086455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086310 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086395 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086441 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086502 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.086634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086532 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rwr9\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.086634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086612 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.086785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086638 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086656 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086673 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.086785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086726 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086802 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086840 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086902 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.086980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086925 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwfpt\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.086980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.086960 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.087223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.087021 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.087223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.087109 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.087223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.087154 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.091888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.091863 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:14:41.092064 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092046 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:14:41.092143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092077 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-5zvln\"" Oct 30 18:14:41.092143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092117 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:14:41.092143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092138 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:14:41.092254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092139 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:14:41.092254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092141 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:14:41.092254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.092161 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:14:41.095815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.095801 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-l5rwl\"" Oct 30 18:14:41.099084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.099063 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:14:41.187727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187705 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.187854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187735 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwfpt\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.187854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187764 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.187965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187870 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.187965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187907 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.187965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187935 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.187970 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188001 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188047 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188075 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188103 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188125 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rwr9\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188168 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188201 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188225 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188255 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188286 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188327 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188357 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.188424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188360 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.188908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.188671 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.189416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.189387 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.189551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.189428 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.191247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.191200 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.191346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.191291 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.191940 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.191890 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.192228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.192203 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.192327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.192270 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.192327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.192319 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.192898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.192860 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.193415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.193367 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.193577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.193556 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.194097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.194074 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.194183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.194105 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.194183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.194079 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.194354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.194335 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.210215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.210192 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwfpt\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:41.211837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.211815 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rwr9\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.295668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.295614 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:41.624561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.624511 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:14:41.628746 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:14:41.628714 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778538ec_a0a3_4f83_b7b7_ecda82081ef9.slice/crio-95242c0a9af102839afd856210a7d0292c674726cb29cbe42a2b8212293b1ab1 WatchSource:0}: Error finding container 95242c0a9af102839afd856210a7d0292c674726cb29cbe42a2b8212293b1ab1: Status 404 returned error can't find the container with id 95242c0a9af102839afd856210a7d0292c674726cb29cbe42a2b8212293b1ab1 Oct 30 18:14:41.941247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:41.941176 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerStarted","Data":"95242c0a9af102839afd856210a7d0292c674726cb29cbe42a2b8212293b1ab1"} Oct 30 18:14:42.948382 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:42.948345 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerStarted","Data":"cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f"} Oct 30 18:14:43.902653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:43.902620 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:14:45.966305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:45.966271 6966 generic.go:356] "Generic (PLEG): container finished" podID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerID="cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f" exitCode=0 Oct 30 18:14:45.966700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:45.966348 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerDied","Data":"cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f"} Oct 30 18:14:46.973521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:46.973492 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerStarted","Data":"9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8"} Oct 30 18:14:46.973932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:46.973557 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerStarted","Data":"99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f"} Oct 30 18:14:46.973932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:46.973740 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:47.040583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.040542 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.855419108 podStartE2EDuration="7.040530605s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.185111497s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:47.028825968 +0000 UTC m=+1049.735011317" watchObservedRunningTime="2026-10-30 18:14:47.040530605 +0000 UTC m=+1049.746715961" Oct 30 18:14:47.198980 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:14:47.198947 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:14:47.438338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.438307 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.464751 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.464725 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.539601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.539569 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.539714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.539692 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.552923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.552902 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.654965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.654936 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.679903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.679883 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:47.906615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:47.906590 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:14:48.245949 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:14:48.245920 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod472e7abd_5671_493b_98cd_0963494e5c00.slice/crio-8dea28b7e120b011c0066fd8bb3a6265143d41884928fafdea6f9b3d7b5bf32e WatchSource:0}: Error finding container 8dea28b7e120b011c0066fd8bb3a6265143d41884928fafdea6f9b3d7b5bf32e: Status 404 returned error can't find the container with id 8dea28b7e120b011c0066fd8bb3a6265143d41884928fafdea6f9b3d7b5bf32e Oct 30 18:14:48.253761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:48.253728 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:14:48.986448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:48.986360 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerStarted","Data":"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30"} Oct 30 18:14:48.986630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:48.986448 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerStarted","Data":"8dea28b7e120b011c0066fd8bb3a6265143d41884928fafdea6f9b3d7b5bf32e"} Oct 30 18:14:52.002721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:52.002693 6966 generic.go:356] "Generic (PLEG): container finished" podID="472e7abd-5671-493b-98cd-0963494e5c00" containerID="0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30" exitCode=0 Oct 30 18:14:52.003078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:52.002741 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerDied","Data":"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30"} Oct 30 18:14:52.981645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:52.981615 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:14:53.010656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:53.010629 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerStarted","Data":"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6"} Oct 30 18:14:53.011019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:53.010683 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerStarted","Data":"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0"} Oct 30 18:14:53.011019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:53.010708 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerStarted","Data":"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735"} Oct 30 18:14:57.907248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:14:57.907215 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:01.111909 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:01.111873 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:02.907786 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:02.907751 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:02.913133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:02.913104 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:02.986030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:02.985980 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=19.796244739 podStartE2EDuration="22.985968279s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.18972354s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:53.09408205 +0000 UTC m=+1055.800267431" watchObservedRunningTime="2026-10-30 18:15:02.985968279 +0000 UTC m=+1065.692153636" Oct 30 18:15:03.067029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:03.067004 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:17.198563 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:17.198526 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:15:18.640107 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:18.640063 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:21.055697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.055665 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c"] Oct 30 18:15:21.056146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.055894 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="kube-rbac-proxy" containerID="cri-o://82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" gracePeriod=30 Oct 30 18:15:21.056146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.056001 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="method-gate" containerID="cri-o://b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" gracePeriod=30 Oct 30 18:15:21.056146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.055989 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="prom-label-proxy" containerID="cri-o://f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" gracePeriod=30 Oct 30 18:15:21.075912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.075885 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg"] Oct 30 18:15:21.076273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.076227 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="kube-rbac-proxy" containerID="cri-o://c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" gracePeriod=30 Oct 30 18:15:21.076273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.076437 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="prom-label-proxy" containerID="cri-o://ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" gracePeriod=30 Oct 30 18:15:21.076679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.076621 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="method-gate" containerID="cri-o://9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" gracePeriod=30 Oct 30 18:15:21.102345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.102321 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm"] Oct 30 18:15:21.102566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.102518 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" podUID="96fd765e-ef8f-4dda-b423-fde2f39c0220" containerName="korrel8r" containerID="cri-o://1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3" gracePeriod=30 Oct 30 18:15:21.361024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.360328 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:15:21.406921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.406880 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:15:21.415283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415253 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config\") pod \"153c539e-4f11-438b-8175-cedabcc2c225\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " Oct 30 18:15:21.415431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415310 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca\") pod \"153c539e-4f11-438b-8175-cedabcc2c225\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " Oct 30 18:15:21.415431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415358 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config\" (UniqueName: \"kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config\") pod \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " Oct 30 18:15:21.415431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415389 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls\" (UniqueName: \"kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls\") pod \"153c539e-4f11-438b-8175-cedabcc2c225\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " Oct 30 18:15:21.415656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415478 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca\") pod \"153c539e-4f11-438b-8175-cedabcc2c225\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " Oct 30 18:15:21.415656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415534 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4\" (UniqueName: \"kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4\") pod \"153c539e-4f11-438b-8175-cedabcc2c225\" (UID: \"153c539e-4f11-438b-8175-cedabcc2c225\") " Oct 30 18:15:21.415656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415573 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls\" (UniqueName: \"kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls\") pod \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " Oct 30 18:15:21.415656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.415617 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc\" (UniqueName: \"kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc\") pod \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\" (UID: \"8dd5a275-5cc7-494f-a922-cddb8f6ff89f\") " Oct 30 18:15:21.417105 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.416878 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca" pod "153c539e-4f11-438b-8175-cedabcc2c225" (UID: "153c539e-4f11-438b-8175-cedabcc2c225"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.417322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.417285 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca" pod "153c539e-4f11-438b-8175-cedabcc2c225" (UID: "153c539e-4f11-438b-8175-cedabcc2c225"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.417407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.417342 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:15:21.417488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.417403 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config" pod "153c539e-4f11-438b-8175-cedabcc2c225" (UID: "153c539e-4f11-438b-8175-cedabcc2c225"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.417676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.417656 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config" pod "8dd5a275-5cc7-494f-a922-cddb8f6ff89f" (UID: "8dd5a275-5cc7-494f-a922-cddb8f6ff89f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.419900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.419877 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls" pod "153c539e-4f11-438b-8175-cedabcc2c225" (UID: "153c539e-4f11-438b-8175-cedabcc2c225"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.419900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.419887 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls" pod "8dd5a275-5cc7-494f-a922-cddb8f6ff89f" (UID: "8dd5a275-5cc7-494f-a922-cddb8f6ff89f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.420498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.420473 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc" pod "8dd5a275-5cc7-494f-a922-cddb8f6ff89f" (UID: "8dd5a275-5cc7-494f-a922-cddb8f6ff89f"). InnerVolumeSpecName "kube-api-access-6pzwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.421278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.421241 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4" pod "153c539e-4f11-438b-8175-cedabcc2c225" (UID: "153c539e-4f11-438b-8175-cedabcc2c225"). InnerVolumeSpecName "kube-api-access-mzzt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.516733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.516696 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config\" (UniqueName: \"kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config\") pod \"96fd765e-ef8f-4dda-b423-fde2f39c0220\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.516785 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls\" (UniqueName: \"kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls\") pod \"96fd765e-ef8f-4dda-b423-fde2f39c0220\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.516850 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs\" (UniqueName: \"kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs\") pod \"96fd765e-ef8f-4dda-b423-fde2f39c0220\" (UID: \"96fd765e-ef8f-4dda-b423-fde2f39c0220\") " Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517162 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517184 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-prometheus-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517202 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517213 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/153c539e-4f11-438b-8175-cedabcc2c225-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517226 6966 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/153c539e-4f11-438b-8175-cedabcc2c225-client-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517241 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzzt4\" (UniqueName: \"kubernetes.io/projected/153c539e-4f11-438b-8175-cedabcc2c225-kube-api-access-mzzt4\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517255 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.517612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517274 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzwc\" (UniqueName: \"kubernetes.io/projected/8dd5a275-5cc7-494f-a922-cddb8f6ff89f-kube-api-access-6pzwc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.518117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.517966 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config" pod "96fd765e-ef8f-4dda-b423-fde2f39c0220" (UID: "96fd765e-ef8f-4dda-b423-fde2f39c0220"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.520237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.520189 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs" pod "96fd765e-ef8f-4dda-b423-fde2f39c0220" (UID: "96fd765e-ef8f-4dda-b423-fde2f39c0220"). InnerVolumeSpecName "kube-api-access-ggvxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.520499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.520455 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls" pod "96fd765e-ef8f-4dda-b423-fde2f39c0220" (UID: "96fd765e-ef8f-4dda-b423-fde2f39c0220"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:21.523009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.522981 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:21.546394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.546366 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:15:21.546638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.546612 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="alertmanager" containerID="cri-o://99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f" gracePeriod=120 Oct 30 18:15:21.546714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.546644 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="config-reloader" containerID="cri-o://9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8" gracePeriod=120 Oct 30 18:15:21.554584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.554556 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:15:21.554889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.554864 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="prometheus" containerID="cri-o://04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" gracePeriod=600 Oct 30 18:15:21.554993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.554885 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="thanos-sidecar" containerID="cri-o://02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" gracePeriod=600 Oct 30 18:15:21.554993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.554886 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="config-reloader" containerID="cri-o://720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" gracePeriod=600 Oct 30 18:15:21.618325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.618271 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggvxs\" (UniqueName: \"kubernetes.io/projected/96fd765e-ef8f-4dda-b423-fde2f39c0220-kube-api-access-ggvxs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.618325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.618293 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/96fd765e-ef8f-4dda-b423-fde2f39c0220-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.618325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.618302 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/96fd765e-ef8f-4dda-b423-fde2f39c0220-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.782996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.782959 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:21.990126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.990098 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:21.990266 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:21.990180 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:22.021917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.021893 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.021942 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022020 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022050 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022082 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022178 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022221 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022250 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022296 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022382 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022422 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out\" (UniqueName: \"kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022477 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.022576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022503 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:22.022576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022512 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"472e7abd-5671-493b-98cd-0963494e5c00\" (UID: \"472e7abd-5671-493b-98cd-0963494e5c00\") " Oct 30 18:15:22.023071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.022800 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:22.023071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.023041 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.023071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.023067 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.023357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.023330 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:22.024574 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.024541 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "kube-api-access-pwfpt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.025293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.025263 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.025400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.025328 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.025400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.025358 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.025637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.025624 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.026038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.026014 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.026141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.026112 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:22.026673 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.026650 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.028187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.028166 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:15:22.037570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.037545 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config" pod "472e7abd-5671-493b-98cd-0963494e5c00" (UID: "472e7abd-5671-493b-98cd-0963494e5c00"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.124273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124206 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124233 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124245 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/472e7abd-5671-493b-98cd-0963494e5c00-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124255 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124265 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124274 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/472e7abd-5671-493b-98cd-0963494e5c00-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124284 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124292 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwfpt\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-kube-api-access-pwfpt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124301 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/472e7abd-5671-493b-98cd-0963494e5c00-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.124770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124328 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:15:22.124770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.124337 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/472e7abd-5671-493b-98cd-0963494e5c00-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.136134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.136115 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:15:22.167228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.167134 6966 generic.go:356] "Generic (PLEG): container finished" podID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerID="9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8" exitCode=0 Oct 30 18:15:22.167356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.167266 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerDied","Data":"9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8"} Oct 30 18:15:22.173043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.173016 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" event={"ID":"96fd765e-ef8f-4dda-b423-fde2f39c0220","Type":"ContainerDied","Data":"1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3"} Oct 30 18:15:22.173043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.173024 6966 generic.go:356] "Generic (PLEG): container finished" podID="96fd765e-ef8f-4dda-b423-fde2f39c0220" containerID="1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3" exitCode=0 Oct 30 18:15:22.173480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.173067 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" event={"ID":"96fd765e-ef8f-4dda-b423-fde2f39c0220","Type":"ContainerDied","Data":"c03f6c11a8a6f4873c92b26409c24fd96fe843f27a17030f4fdb5d972f123c17"} Oct 30 18:15:22.173480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.173074 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm" Oct 30 18:15:22.173480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.173089 6966 scope.go:122] "RemoveContainer" containerID="1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3" Oct 30 18:15:22.175155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175130 6966 generic.go:356] "Generic (PLEG): container finished" podID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerID="ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" exitCode=0 Oct 30 18:15:22.175155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175154 6966 generic.go:356] "Generic (PLEG): container finished" podID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerID="9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" exitCode=0 Oct 30 18:15:22.175334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175164 6966 generic.go:356] "Generic (PLEG): container finished" podID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerID="c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" exitCode=0 Oct 30 18:15:22.175334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175196 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" Oct 30 18:15:22.175334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175197 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerDied","Data":"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2"} Oct 30 18:15:22.175334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175251 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerDied","Data":"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5"} Oct 30 18:15:22.175334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175277 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerDied","Data":"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a"} Oct 30 18:15:22.175334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.175305 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg" event={"ID":"8dd5a275-5cc7-494f-a922-cddb8f6ff89f","Type":"ContainerDied","Data":"6112b28aff7b26e3502a8f7bef299b9fc58718c66711483b1955362ad6842af6"} Oct 30 18:15:22.180035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180011 6966 generic.go:356] "Generic (PLEG): container finished" podID="472e7abd-5671-493b-98cd-0963494e5c00" containerID="02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" exitCode=0 Oct 30 18:15:22.180035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180034 6966 generic.go:356] "Generic (PLEG): container finished" podID="472e7abd-5671-493b-98cd-0963494e5c00" containerID="720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" exitCode=0 Oct 30 18:15:22.180183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180044 6966 generic.go:356] "Generic (PLEG): container finished" podID="472e7abd-5671-493b-98cd-0963494e5c00" containerID="04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" exitCode=0 Oct 30 18:15:22.180183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180078 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:15:22.180183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180088 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerDied","Data":"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6"} Oct 30 18:15:22.180183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180163 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerDied","Data":"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0"} Oct 30 18:15:22.180395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180203 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerDied","Data":"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735"} Oct 30 18:15:22.180395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.180242 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"472e7abd-5671-493b-98cd-0963494e5c00","Type":"ContainerDied","Data":"8dea28b7e120b011c0066fd8bb3a6265143d41884928fafdea6f9b3d7b5bf32e"} Oct 30 18:15:22.184196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184130 6966 scope.go:122] "RemoveContainer" containerID="1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3" Oct 30 18:15:22.184769 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.184740 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3\": container with ID starting with 1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3 not found: ID does not exist" containerID="1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3" Oct 30 18:15:22.184769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184759 6966 generic.go:356] "Generic (PLEG): container finished" podID="153c539e-4f11-438b-8175-cedabcc2c225" containerID="f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" exitCode=0 Oct 30 18:15:22.184933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184777 6966 generic.go:356] "Generic (PLEG): container finished" podID="153c539e-4f11-438b-8175-cedabcc2c225" containerID="b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" exitCode=0 Oct 30 18:15:22.184933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184778 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3"} err="failed to get container status \"1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3\": rpc error: code = NotFound desc = could not find container \"1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3\": container with ID starting with 1d7b6a97a65d557227d213cbe4ca463fad162166e690925d2615e36086814fc3 not found: ID does not exist" Oct 30 18:15:22.184933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184801 6966 scope.go:122] "RemoveContainer" containerID="ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" Oct 30 18:15:22.184933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184790 6966 generic.go:356] "Generic (PLEG): container finished" podID="153c539e-4f11-438b-8175-cedabcc2c225" containerID="82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" exitCode=0 Oct 30 18:15:22.184933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184798 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerDied","Data":"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d"} Oct 30 18:15:22.184933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184905 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" Oct 30 18:15:22.185270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.184965 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerDied","Data":"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df"} Oct 30 18:15:22.185270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.185013 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerDied","Data":"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af"} Oct 30 18:15:22.185270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.185052 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c" event={"ID":"153c539e-4f11-438b-8175-cedabcc2c225","Type":"ContainerDied","Data":"6c47e8ba6a1ca58b0c52d1d00b9f4bd915876b7564a2027d15ee971ac78ede72"} Oct 30 18:15:22.194237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.194220 6966 scope.go:122] "RemoveContainer" containerID="9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" Oct 30 18:15:22.202041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.202027 6966 scope.go:122] "RemoveContainer" containerID="c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" Oct 30 18:15:22.209336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.209320 6966 scope.go:122] "RemoveContainer" containerID="ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" Oct 30 18:15:22.209609 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.209590 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": container with ID starting with ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2 not found: ID does not exist" containerID="ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" Oct 30 18:15:22.209684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.209618 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2"} err="failed to get container status \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": rpc error: code = NotFound desc = could not find container \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": container with ID starting with ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2 not found: ID does not exist" Oct 30 18:15:22.209684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.209638 6966 scope.go:122] "RemoveContainer" containerID="9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" Oct 30 18:15:22.209897 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.209880 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": container with ID starting with 9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5 not found: ID does not exist" containerID="9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" Oct 30 18:15:22.209935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.209901 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5"} err="failed to get container status \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": rpc error: code = NotFound desc = could not find container \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": container with ID starting with 9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5 not found: ID does not exist" Oct 30 18:15:22.209935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.209916 6966 scope.go:122] "RemoveContainer" containerID="c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" Oct 30 18:15:22.210150 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.210133 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": container with ID starting with c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a not found: ID does not exist" containerID="c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" Oct 30 18:15:22.210197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210155 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a"} err="failed to get container status \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": rpc error: code = NotFound desc = could not find container \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": container with ID starting with c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a not found: ID does not exist" Oct 30 18:15:22.210197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210170 6966 scope.go:122] "RemoveContainer" containerID="ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" Oct 30 18:15:22.210405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210388 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2"} err="failed to get container status \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": rpc error: code = NotFound desc = could not find container \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": container with ID starting with ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2 not found: ID does not exist" Oct 30 18:15:22.210604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210407 6966 scope.go:122] "RemoveContainer" containerID="9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" Oct 30 18:15:22.210674 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210656 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5"} err="failed to get container status \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": rpc error: code = NotFound desc = could not find container \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": container with ID starting with 9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5 not found: ID does not exist" Oct 30 18:15:22.210738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210676 6966 scope.go:122] "RemoveContainer" containerID="c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" Oct 30 18:15:22.210963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210930 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a"} err="failed to get container status \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": rpc error: code = NotFound desc = could not find container \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": container with ID starting with c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a not found: ID does not exist" Oct 30 18:15:22.211011 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.210965 6966 scope.go:122] "RemoveContainer" containerID="ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2" Oct 30 18:15:22.211207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.211189 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2"} err="failed to get container status \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": rpc error: code = NotFound desc = could not find container \"ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2\": container with ID starting with ba8908c35029c1941c7797f226f60c57f5f7922d1ee7ee83e0038f08329395c2 not found: ID does not exist" Oct 30 18:15:22.211248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.211208 6966 scope.go:122] "RemoveContainer" containerID="9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5" Oct 30 18:15:22.211402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.211384 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5"} err="failed to get container status \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": rpc error: code = NotFound desc = could not find container \"9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5\": container with ID starting with 9fd257b3fce3ae8ba271d7c84b24a52a51746080a5e0bfb569d52379d4d11ef5 not found: ID does not exist" Oct 30 18:15:22.211486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.211403 6966 scope.go:122] "RemoveContainer" containerID="c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a" Oct 30 18:15:22.211646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.211627 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a"} err="failed to get container status \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": rpc error: code = NotFound desc = could not find container \"c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a\": container with ID starting with c6d327a01dde319c7cec2ed36bf1cc2023a0229faa591781c723cc6074008e2a not found: ID does not exist" Oct 30 18:15:22.211686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.211647 6966 scope.go:122] "RemoveContainer" containerID="02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" Oct 30 18:15:22.219042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.219025 6966 scope.go:122] "RemoveContainer" containerID="720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" Oct 30 18:15:22.223263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.223240 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm"] Oct 30 18:15:22.224947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.224917 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 18:15:22.228211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.228188 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-mf4qm"] Oct 30 18:15:22.230733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.230026 6966 scope.go:122] "RemoveContainer" containerID="04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" Oct 30 18:15:22.241370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.241352 6966 scope.go:122] "RemoveContainer" containerID="0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30" Oct 30 18:15:22.253715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.253699 6966 scope.go:122] "RemoveContainer" containerID="02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" Oct 30 18:15:22.254141 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.254116 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": container with ID starting with 02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6 not found: ID does not exist" containerID="02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" Oct 30 18:15:22.254232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.254154 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6"} err="failed to get container status \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": rpc error: code = NotFound desc = could not find container \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": container with ID starting with 02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6 not found: ID does not exist" Oct 30 18:15:22.254232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.254176 6966 scope.go:122] "RemoveContainer" containerID="720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" Oct 30 18:15:22.254523 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.254491 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": container with ID starting with 720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0 not found: ID does not exist" containerID="720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" Oct 30 18:15:22.254619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.254534 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0"} err="failed to get container status \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": rpc error: code = NotFound desc = could not find container \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": container with ID starting with 720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0 not found: ID does not exist" Oct 30 18:15:22.254619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.254560 6966 scope.go:122] "RemoveContainer" containerID="04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" Oct 30 18:15:22.254892 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.254867 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": container with ID starting with 04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735 not found: ID does not exist" containerID="04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" Oct 30 18:15:22.254957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.254903 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735"} err="failed to get container status \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": rpc error: code = NotFound desc = could not find container \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": container with ID starting with 04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735 not found: ID does not exist" Oct 30 18:15:22.254957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.254925 6966 scope.go:122] "RemoveContainer" containerID="0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30" Oct 30 18:15:22.255177 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.255157 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": container with ID starting with 0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30 not found: ID does not exist" containerID="0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30" Oct 30 18:15:22.255255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255180 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30"} err="failed to get container status \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": rpc error: code = NotFound desc = could not find container \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": container with ID starting with 0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30 not found: ID does not exist" Oct 30 18:15:22.255255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255198 6966 scope.go:122] "RemoveContainer" containerID="02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" Oct 30 18:15:22.255446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255425 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6"} err="failed to get container status \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": rpc error: code = NotFound desc = could not find container \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": container with ID starting with 02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6 not found: ID does not exist" Oct 30 18:15:22.255529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255445 6966 scope.go:122] "RemoveContainer" containerID="720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" Oct 30 18:15:22.255695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255676 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0"} err="failed to get container status \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": rpc error: code = NotFound desc = could not find container \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": container with ID starting with 720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0 not found: ID does not exist" Oct 30 18:15:22.255695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255693 6966 scope.go:122] "RemoveContainer" containerID="04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" Oct 30 18:15:22.255932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255907 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735"} err="failed to get container status \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": rpc error: code = NotFound desc = could not find container \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": container with ID starting with 04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735 not found: ID does not exist" Oct 30 18:15:22.256001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.255932 6966 scope.go:122] "RemoveContainer" containerID="0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30" Oct 30 18:15:22.256171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256150 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30"} err="failed to get container status \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": rpc error: code = NotFound desc = could not find container \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": container with ID starting with 0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30 not found: ID does not exist" Oct 30 18:15:22.256240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256171 6966 scope.go:122] "RemoveContainer" containerID="02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6" Oct 30 18:15:22.256394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256368 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6"} err="failed to get container status \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": rpc error: code = NotFound desc = could not find container \"02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6\": container with ID starting with 02282d3f4b61f19e3ce8d6703471f621da28a19c7ebd1504c92b9c30665ca3b6 not found: ID does not exist" Oct 30 18:15:22.256477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256395 6966 scope.go:122] "RemoveContainer" containerID="720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0" Oct 30 18:15:22.256656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256628 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0"} err="failed to get container status \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": rpc error: code = NotFound desc = could not find container \"720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0\": container with ID starting with 720f7c207a553aa5a88bb4d32f36cf4ee26789a390220f4ba766c7ab9d8e88d0 not found: ID does not exist" Oct 30 18:15:22.256755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256657 6966 scope.go:122] "RemoveContainer" containerID="04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735" Oct 30 18:15:22.256923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256899 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735"} err="failed to get container status \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": rpc error: code = NotFound desc = could not find container \"04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735\": container with ID starting with 04a5627e60ce095942accbc98e4c82f5ee04e2149a914a8c8e1f775b8fc2b735 not found: ID does not exist" Oct 30 18:15:22.257012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.256925 6966 scope.go:122] "RemoveContainer" containerID="0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30" Oct 30 18:15:22.257152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.257127 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30"} err="failed to get container status \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": rpc error: code = NotFound desc = could not find container \"0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30\": container with ID starting with 0323e0d47e0d0b62872fe1b3907023f4b09a72493080666c6c61f804c1723f30 not found: ID does not exist" Oct 30 18:15:22.257235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.257150 6966 scope.go:122] "RemoveContainer" containerID="f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" Oct 30 18:15:22.272653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.272632 6966 scope.go:122] "RemoveContainer" containerID="b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" Oct 30 18:15:22.275605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.275580 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:15:22.275824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.275799 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="125e9d51-9af0-4237-916d-8cba28e3f7a0" containerName="otc-container" containerID="cri-o://78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417" gracePeriod=30 Oct 30 18:15:22.289398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.289378 6966 scope.go:122] "RemoveContainer" containerID="82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" Oct 30 18:15:22.300584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.300451 6966 scope.go:122] "RemoveContainer" containerID="f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" Oct 30 18:15:22.300959 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.300932 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": container with ID starting with f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d not found: ID does not exist" containerID="f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" Oct 30 18:15:22.301060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.300971 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d"} err="failed to get container status \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": rpc error: code = NotFound desc = could not find container \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": container with ID starting with f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d not found: ID does not exist" Oct 30 18:15:22.301060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.300994 6966 scope.go:122] "RemoveContainer" containerID="b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" Oct 30 18:15:22.301295 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.301266 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": container with ID starting with b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df not found: ID does not exist" containerID="b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" Oct 30 18:15:22.301359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.301299 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df"} err="failed to get container status \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": rpc error: code = NotFound desc = could not find container \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": container with ID starting with b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df not found: ID does not exist" Oct 30 18:15:22.301359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.301318 6966 scope.go:122] "RemoveContainer" containerID="82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" Oct 30 18:15:22.301694 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:22.301671 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": container with ID starting with 82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af not found: ID does not exist" containerID="82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" Oct 30 18:15:22.301776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.301701 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af"} err="failed to get container status \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": rpc error: code = NotFound desc = could not find container \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": container with ID starting with 82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af not found: ID does not exist" Oct 30 18:15:22.301776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.301725 6966 scope.go:122] "RemoveContainer" containerID="f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" Oct 30 18:15:22.302017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.301991 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d"} err="failed to get container status \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": rpc error: code = NotFound desc = could not find container \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": container with ID starting with f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d not found: ID does not exist" Oct 30 18:15:22.302080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302019 6966 scope.go:122] "RemoveContainer" containerID="b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" Oct 30 18:15:22.302342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302319 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df"} err="failed to get container status \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": rpc error: code = NotFound desc = could not find container \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": container with ID starting with b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df not found: ID does not exist" Oct 30 18:15:22.302400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302344 6966 scope.go:122] "RemoveContainer" containerID="82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" Oct 30 18:15:22.302633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302606 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af"} err="failed to get container status \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": rpc error: code = NotFound desc = could not find container \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": container with ID starting with 82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af not found: ID does not exist" Oct 30 18:15:22.302633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302632 6966 scope.go:122] "RemoveContainer" containerID="f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d" Oct 30 18:15:22.302902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302880 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d"} err="failed to get container status \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": rpc error: code = NotFound desc = could not find container \"f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d\": container with ID starting with f7220156d102454e732ed0b6109998878664089bac736ce65322f88ce0da6d5d not found: ID does not exist" Oct 30 18:15:22.302981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.302903 6966 scope.go:122] "RemoveContainer" containerID="b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df" Oct 30 18:15:22.303218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.303184 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df"} err="failed to get container status \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": rpc error: code = NotFound desc = could not find container \"b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df\": container with ID starting with b97787648df476da9dcd9f5531205dc60bae6ca591ed5931f5091de6d612f3df not found: ID does not exist" Oct 30 18:15:22.303218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.303210 6966 scope.go:122] "RemoveContainer" containerID="82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af" Oct 30 18:15:22.303531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.303508 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af"} err="failed to get container status \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": rpc error: code = NotFound desc = could not find container \"82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af\": container with ID starting with 82d6986dc5b29869cd56dc69d2d6054fb28eec4fc2fe7f3b520c8136d72ee6af not found: ID does not exist" Oct 30 18:15:22.314415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.314391 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c"] Oct 30 18:15:22.319313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.319266 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-rlt9c"] Oct 30 18:15:22.354860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.354831 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:15:22.359436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.359413 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:15:22.391175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.391113 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg"] Oct 30 18:15:22.394992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.394965 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:22.395970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.395918 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-h6cxg"] Oct 30 18:15:22.525246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.525225 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:22.599614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.599577 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:22.628644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.628614 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5\" (UniqueName: \"kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5\") pod \"125e9d51-9af0-4237-916d-8cba28e3f7a0\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " Oct 30 18:15:22.628809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.628657 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs\") pod \"125e9d51-9af0-4237-916d-8cba28e3f7a0\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " Oct 30 18:15:22.628809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.628682 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs\") pod \"125e9d51-9af0-4237-916d-8cba28e3f7a0\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " Oct 30 18:15:22.628921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.628810 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs\") pod \"125e9d51-9af0-4237-916d-8cba28e3f7a0\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " Oct 30 18:15:22.628921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.628865 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert\") pod \"125e9d51-9af0-4237-916d-8cba28e3f7a0\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " Oct 30 18:15:22.628921 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.628915 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal\" (UniqueName: \"kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal\") pod \"125e9d51-9af0-4237-916d-8cba28e3f7a0\" (UID: \"125e9d51-9af0-4237-916d-8cba28e3f7a0\") " Oct 30 18:15:22.629540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.629512 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal" pod "125e9d51-9af0-4237-916d-8cba28e3f7a0" (UID: "125e9d51-9af0-4237-916d-8cba28e3f7a0"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:22.631448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.631421 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs" pod "125e9d51-9af0-4237-916d-8cba28e3f7a0" (UID: "125e9d51-9af0-4237-916d-8cba28e3f7a0"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.631586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.631445 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert" pod "125e9d51-9af0-4237-916d-8cba28e3f7a0" (UID: "125e9d51-9af0-4237-916d-8cba28e3f7a0"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.631586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.631511 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs" pod "125e9d51-9af0-4237-916d-8cba28e3f7a0" (UID: "125e9d51-9af0-4237-916d-8cba28e3f7a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.631586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.631527 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5" pod "125e9d51-9af0-4237-916d-8cba28e3f7a0" (UID: "125e9d51-9af0-4237-916d-8cba28e3f7a0"). InnerVolumeSpecName "kube-api-access-b2xk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.632082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.632057 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs" pod "125e9d51-9af0-4237-916d-8cba28e3f7a0" (UID: "125e9d51-9af0-4237-916d-8cba28e3f7a0"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:22.730180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.730122 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.730180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.730148 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.730180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.730158 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/125e9d51-9af0-4237-916d-8cba28e3f7a0-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.730180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.730167 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xk5\" (UniqueName: \"kubernetes.io/projected/125e9d51-9af0-4237-916d-8cba28e3f7a0-kube-api-access-b2xk5\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.730180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.730175 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.730419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.730184 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/125e9d51-9af0-4237-916d-8cba28e3f7a0-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.980740 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:22.980668 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.58:9093/-/ready\": dial tcp 10.134.0.58:9093: connect: connection refused" Oct 30 18:15:23.202851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.202818 6966 generic.go:356] "Generic (PLEG): container finished" podID="125e9d51-9af0-4237-916d-8cba28e3f7a0" containerID="78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417" exitCode=0 Oct 30 18:15:23.203296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.202870 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"125e9d51-9af0-4237-916d-8cba28e3f7a0","Type":"ContainerDied","Data":"78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417"} Oct 30 18:15:23.203296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.202886 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:23.203296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.202923 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"125e9d51-9af0-4237-916d-8cba28e3f7a0","Type":"ContainerDied","Data":"f4f3587a89b188c3790c1de7016b54272958664a4127b7b8ffa5a536c4e0156d"} Oct 30 18:15:23.203296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.202968 6966 scope.go:122] "RemoveContainer" containerID="78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417" Oct 30 18:15:23.214059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.214041 6966 scope.go:122] "RemoveContainer" containerID="78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417" Oct 30 18:15:23.214527 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:23.214492 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417\": container with ID starting with 78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417 not found: ID does not exist" containerID="78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417" Oct 30 18:15:23.214649 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.214533 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417"} err="failed to get container status \"78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417\": rpc error: code = NotFound desc = could not find container \"78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417\": container with ID starting with 78cc38ef1e9b352539050325e9810d4e184c21be15d20357a4f0db6e17507417 not found: ID does not exist" Oct 30 18:15:23.215006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.214970 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:23.254113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.254045 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:15:23.258283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.258255 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:15:23.412372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.412340 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="125e9d51-9af0-4237-916d-8cba28e3f7a0" path="/var/lib/kubelet/pods/125e9d51-9af0-4237-916d-8cba28e3f7a0/volumes" Oct 30 18:15:23.412791 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.412773 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="153c539e-4f11-438b-8175-cedabcc2c225" path="/var/lib/kubelet/pods/153c539e-4f11-438b-8175-cedabcc2c225/volumes" Oct 30 18:15:23.413307 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.413293 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="472e7abd-5671-493b-98cd-0963494e5c00" path="/var/lib/kubelet/pods/472e7abd-5671-493b-98cd-0963494e5c00/volumes" Oct 30 18:15:23.414141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.414126 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" path="/var/lib/kubelet/pods/8dd5a275-5cc7-494f-a922-cddb8f6ff89f/volumes" Oct 30 18:15:23.414583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.414570 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96fd765e-ef8f-4dda-b423-fde2f39c0220" path="/var/lib/kubelet/pods/96fd765e-ef8f-4dda-b423-fde2f39c0220/volumes" Oct 30 18:15:23.427662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.427631 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:23.657955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:23.657927 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:27.981113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:27.981072 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.58:9093/-/ready\": dial tcp 10.134.0.58:9093: connect: connection refused" Oct 30 18:15:31.786708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.786684 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:31.899114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899088 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899125 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899161 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899180 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899244 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899272 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899301 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config\") pod \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\" (UID: \"778538ec-a0a3-4f83-b7b7-ecda82081ef9\") " Oct 30 18:15:31.899645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.899593 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:31.901403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.901370 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "kube-api-access-2rwr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:31.901862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.901839 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:31.902305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.902277 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:31.902406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.902380 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:15:31.905891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.905869 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:31.912508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:31.912487 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config" pod "778538ec-a0a3-4f83-b7b7-ecda82081ef9" (UID: "778538ec-a0a3-4f83-b7b7-ecda82081ef9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:32.000170 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000145 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.000267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000186 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.000267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000196 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rwr9\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-kube-api-access-2rwr9\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.000267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000208 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.000267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000217 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.000267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000225 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/778538ec-a0a3-4f83-b7b7-ecda82081ef9-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.000267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.000233 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/778538ec-a0a3-4f83-b7b7-ecda82081ef9-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.230675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.230594 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:32.251053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.251021 6966 generic.go:356] "Generic (PLEG): container finished" podID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerID="99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f" exitCode=0 Oct 30 18:15:32.251206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.251079 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:15:32.251206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.251102 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerDied","Data":"99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f"} Oct 30 18:15:32.251206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.251181 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"778538ec-a0a3-4f83-b7b7-ecda82081ef9","Type":"ContainerDied","Data":"95242c0a9af102839afd856210a7d0292c674726cb29cbe42a2b8212293b1ab1"} Oct 30 18:15:32.251322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.251228 6966 scope.go:122] "RemoveContainer" containerID="9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8" Oct 30 18:15:32.267010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.266994 6966 scope.go:122] "RemoveContainer" containerID="99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f" Oct 30 18:15:32.274642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.274619 6966 scope.go:122] "RemoveContainer" containerID="cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f" Oct 30 18:15:32.283145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.283122 6966 scope.go:122] "RemoveContainer" containerID="9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8" Oct 30 18:15:32.283402 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:32.283381 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8\": container with ID starting with 9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8 not found: ID does not exist" containerID="9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8" Oct 30 18:15:32.283451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.283411 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8"} err="failed to get container status \"9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8\": rpc error: code = NotFound desc = could not find container \"9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8\": container with ID starting with 9daf6ee66e2841f506cfdcf8b566c27745466b2b9859d3d53d0ea5e60a96b4c8 not found: ID does not exist" Oct 30 18:15:32.283451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.283428 6966 scope.go:122] "RemoveContainer" containerID="99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f" Oct 30 18:15:32.283712 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:32.283693 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f\": container with ID starting with 99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f not found: ID does not exist" containerID="99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f" Oct 30 18:15:32.283756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.283723 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f"} err="failed to get container status \"99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f\": rpc error: code = NotFound desc = could not find container \"99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f\": container with ID starting with 99386b663da700b31c0401fbc2e8684c2f96ec0a1328271e9b6ddb349b52fc9f not found: ID does not exist" Oct 30 18:15:32.283756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.283740 6966 scope.go:122] "RemoveContainer" containerID="cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f" Oct 30 18:15:32.283973 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:32.283957 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f\": container with ID starting with cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f not found: ID does not exist" containerID="cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f" Oct 30 18:15:32.284019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.283978 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f"} err="failed to get container status \"cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f\": rpc error: code = NotFound desc = could not find container \"cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f\": container with ID starting with cf34503ec28fc2038edd9e3318b03d36ed9a9a930178f2c5fdfa3ef9c904fd4f not found: ID does not exist" Oct 30 18:15:32.301916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.301895 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:15:32.305157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.305132 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:15:32.456877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.456848 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l"] Oct 30 18:15:32.457173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457161 6966 cpu_manager.go:414] "containerMap: removing container" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="prom-label-proxy" Oct 30 18:15:32.457216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457174 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="prom-label-proxy" Oct 30 18:15:32.457216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457188 6966 cpu_manager.go:414] "containerMap: removing container" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="kube-rbac-proxy" Oct 30 18:15:32.457216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457193 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="kube-rbac-proxy" Oct 30 18:15:32.457216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457207 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="method-gate" Oct 30 18:15:32.457216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457213 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="method-gate" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457222 6966 cpu_manager.go:414] "containerMap: removing container" podUID="96fd765e-ef8f-4dda-b423-fde2f39c0220" containerName="korrel8r" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457227 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96fd765e-ef8f-4dda-b423-fde2f39c0220" containerName="korrel8r" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457233 6966 cpu_manager.go:414] "containerMap: removing container" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457238 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457244 6966 cpu_manager.go:414] "containerMap: removing container" podUID="125e9d51-9af0-4237-916d-8cba28e3f7a0" containerName="otc-container" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457249 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="125e9d51-9af0-4237-916d-8cba28e3f7a0" containerName="otc-container" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457259 6966 cpu_manager.go:414] "containerMap: removing container" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457264 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457271 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="prom-label-proxy" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457275 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="prom-label-proxy" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457282 6966 cpu_manager.go:414] "containerMap: removing container" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="init-config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457286 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="init-config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457292 6966 cpu_manager.go:414] "containerMap: removing container" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="alertmanager" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457296 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="alertmanager" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457305 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="kube-rbac-proxy" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457309 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="kube-rbac-proxy" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457318 6966 cpu_manager.go:414] "containerMap: removing container" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="prometheus" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457323 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="prometheus" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457334 6966 cpu_manager.go:414] "containerMap: removing container" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="method-gate" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457338 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="method-gate" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457345 6966 cpu_manager.go:414] "containerMap: removing container" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="thanos-sidecar" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457349 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="thanos-sidecar" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457356 6966 cpu_manager.go:414] "containerMap: removing container" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="init-config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457360 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="init-config-reloader" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457418 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="method-gate" Oct 30 18:15:32.457413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457427 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="96fd765e-ef8f-4dda-b423-fde2f39c0220" containerName="korrel8r" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457435 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="config-reloader" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457442 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="125e9d51-9af0-4237-916d-8cba28e3f7a0" containerName="otc-container" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457448 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="config-reloader" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457454 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="prom-label-proxy" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457473 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" containerName="alertmanager" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457481 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dd5a275-5cc7-494f-a922-cddb8f6ff89f" containerName="kube-rbac-proxy" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457489 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="prometheus" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457495 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="method-gate" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457500 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="472e7abd-5671-493b-98cd-0963494e5c00" containerName="thanos-sidecar" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457505 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="prom-label-proxy" Oct 30 18:15:32.458564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.457511 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="153c539e-4f11-438b-8175-cedabcc2c225" containerName="kube-rbac-proxy" Oct 30 18:15:32.459839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.459817 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:32.462638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.462618 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.481907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.481855 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l"] Oct 30 18:15:32.499081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.499060 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l"] Oct 30 18:15:32.502777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.502760 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.505325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.505301 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.505422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.505372 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.505485 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.505441 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2n6\" (UniqueName: \"kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.505551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.505536 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.505596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.505585 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.542005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.541985 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:15:32.542592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.542575 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:15:32.542657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.542579 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:15:32.542657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.542611 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:15:32.543766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.543750 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:15:32.543838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.543822 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:15:32.543980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.543956 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:15:32.544098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.544005 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-vv7lj\"" Oct 30 18:15:32.565645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.565627 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l"] Oct 30 18:15:32.606617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606592 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.606716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606623 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.606716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606647 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgs8p\" (UniqueName: \"kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.606792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606768 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.606826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606805 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2n6\" (UniqueName: \"kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.606898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606882 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.606946 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606916 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.607000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.606960 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.607280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.607265 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.607319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.607284 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.607585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.607569 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.609264 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.609244 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.623092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.623073 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx"] Oct 30 18:15:32.626835 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.626819 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.640862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.640835 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:15:32.640862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.640850 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-bt7fr\"" Oct 30 18:15:32.640862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.640843 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-88q9w\"" Oct 30 18:15:32.641034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.640891 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:15:32.641072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.641047 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:15:32.643065 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.643043 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2n6\" (UniqueName: \"kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-77x5l\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.649056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.649036 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx"] Oct 30 18:15:32.708292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708269 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.708377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708321 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.708436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708418 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.708515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708454 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.708557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708512 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgs8p\" (UniqueName: \"kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.708595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708564 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.708957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.708935 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.710676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.710658 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.713366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.713345 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:32.731584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.731559 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgs8p\" (UniqueName: \"kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6c42l\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.779852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.779795 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:15:32.810035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.810007 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.810397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.810054 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.810397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.810165 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.810979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.810939 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.812411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.812385 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.820321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.820303 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:15:32.835828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.835806 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp\") pod \"korrel8r-5cbcc89b8f-ssfwx\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:32.936934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.936901 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:32.939782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:32.939744 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:33.081202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.081023 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l"] Oct 30 18:15:33.084716 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:15:33.084684 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd012103e_2215_41a0_8217_61bb33ffa01a.slice/crio-d47a82e46f7a318233221b54fd2298bbf57eccf16a1b250a1edda762795f525a WatchSource:0}: Error finding container d47a82e46f7a318233221b54fd2298bbf57eccf16a1b250a1edda762795f525a: Status 404 returned error can't find the container with id d47a82e46f7a318233221b54fd2298bbf57eccf16a1b250a1edda762795f525a Oct 30 18:15:33.125728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.125704 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l"] Oct 30 18:15:33.130125 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:15:33.130102 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eb340d_c73d_4ede_90eb_c17b4f595130.slice/crio-3a49524b10b453bc171208fb1217c3ac75e3e41f769b6702690b46822a19508f WatchSource:0}: Error finding container 3a49524b10b453bc171208fb1217c3ac75e3e41f769b6702690b46822a19508f: Status 404 returned error can't find the container with id 3a49524b10b453bc171208fb1217c3ac75e3e41f769b6702690b46822a19508f Oct 30 18:15:33.224684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.224061 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:33.236505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.235054 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx"] Oct 30 18:15:33.240235 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:15:33.240206 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c5be6c1_a1b5_45f0_8ca0_8ab4e47b274f.slice/crio-4391d53a2a484c0c7cc7619f5a3b9b4a119d0e779a23ed72ee7643b65f35c5ba WatchSource:0}: Error finding container 4391d53a2a484c0c7cc7619f5a3b9b4a119d0e779a23ed72ee7643b65f35c5ba: Status 404 returned error can't find the container with id 4391d53a2a484c0c7cc7619f5a3b9b4a119d0e779a23ed72ee7643b65f35c5ba Oct 30 18:15:33.268238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.268212 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" event={"ID":"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f","Type":"ContainerStarted","Data":"4391d53a2a484c0c7cc7619f5a3b9b4a119d0e779a23ed72ee7643b65f35c5ba"} Oct 30 18:15:33.270634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.270604 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerStarted","Data":"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4"} Oct 30 18:15:33.270735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.270676 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerStarted","Data":"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec"} Oct 30 18:15:33.270735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.270714 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerStarted","Data":"3a49524b10b453bc171208fb1217c3ac75e3e41f769b6702690b46822a19508f"} Oct 30 18:15:33.272965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.272941 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerStarted","Data":"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff"} Oct 30 18:15:33.273068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.272990 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerStarted","Data":"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf"} Oct 30 18:15:33.273068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.273014 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerStarted","Data":"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f"} Oct 30 18:15:33.273068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.273040 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerStarted","Data":"d47a82e46f7a318233221b54fd2298bbf57eccf16a1b250a1edda762795f525a"} Oct 30 18:15:33.316749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.316705 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" podStartSLOduration=1.316693271 podStartE2EDuration="1.316693271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:33.314878428 +0000 UTC m=+1096.021063786" watchObservedRunningTime="2026-10-30 18:15:33.316693271 +0000 UTC m=+1096.022878624" Oct 30 18:15:33.411512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.411416 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778538ec-a0a3-4f83-b7b7-ecda82081ef9" path="/var/lib/kubelet/pods/778538ec-a0a3-4f83-b7b7-ecda82081ef9/volumes" Oct 30 18:15:33.447717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.447661 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:33.665013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:33.664933 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:34.081595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.081562 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:15:34.085948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.085925 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.103936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.103915 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:15:34.105468 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.105435 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:15:34.122743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.122719 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.122842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.122762 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.122842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.122790 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.122955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.122934 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.122990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.122975 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.123065 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.123047 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvr5\" (UniqueName: \"kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.169987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.169951 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:15:34.169987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.169970 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:15:34.170204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.170005 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:15:34.170204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.170023 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:15:34.170204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.170056 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:15:34.189354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.189325 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:34.223497 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.223447 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.223635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.223510 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.223635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.223539 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.223635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.223556 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.223635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.223584 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvr5\" (UniqueName: \"kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.223849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.223646 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.223849 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:34.223680 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 18:15:34.223849 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:34.223768 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs podName:ad20e6e1-68cf-4caf-9f5c-0a36309cbd76 nodeName:}" failed. No retries permitted until 2026-10-30 18:15:34.72374767 +0000 UTC m=+1097.429933019 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76") : secret "data-science-collector-collector-headless-tls" not found Oct 30 18:15:34.224221 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.224204 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.226003 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.225982 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.226089 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.226013 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.226089 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.226032 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.249087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.249067 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvr5\" (UniqueName: \"kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.279971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.279943 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerStarted","Data":"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde"} Oct 30 18:15:34.281699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.281668 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" event={"ID":"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f","Type":"ContainerStarted","Data":"31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a"} Oct 30 18:15:34.282001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.281985 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:34.322606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.322567 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" podStartSLOduration=2.322554849 podStartE2EDuration="2.322554849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:34.319825959 +0000 UTC m=+1097.026011318" watchObservedRunningTime="2026-10-30 18:15:34.322554849 +0000 UTC m=+1097.028740205" Oct 30 18:15:34.359790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.359704 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" podStartSLOduration=2.359692111 podStartE2EDuration="2.359692111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:34.35800994 +0000 UTC m=+1097.064195297" watchObservedRunningTime="2026-10-30 18:15:34.359692111 +0000 UTC m=+1097.065877468" Oct 30 18:15:34.409429 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.409404 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:34.626202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.626130 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:34.727999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.727970 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:34.730321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:34.730303 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:35.000298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:35.000219 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:15:35.299383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:35.299314 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:15:35.304552 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:15:35.304523 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad20e6e1_68cf_4caf_9f5c_0a36309cbd76.slice/crio-ad860800dda8071684e5c8add92f32dcd2c8a665d976ef6779f1d8f46722d1b1 WatchSource:0}: Error finding container ad860800dda8071684e5c8add92f32dcd2c8a665d976ef6779f1d8f46722d1b1: Status 404 returned error can't find the container with id ad860800dda8071684e5c8add92f32dcd2c8a665d976ef6779f1d8f46722d1b1 Oct 30 18:15:35.306960 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:35.306922 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:15:35.307053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:35.307029 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:15:36.294387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:36.294353 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76","Type":"ContainerStarted","Data":"4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602"} Oct 30 18:15:36.294580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:36.294411 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76","Type":"ContainerStarted","Data":"ad860800dda8071684e5c8add92f32dcd2c8a665d976ef6779f1d8f46722d1b1"} Oct 30 18:15:36.346456 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:36.346396 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.346378869 podStartE2EDuration="2.346378869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:36.334997203 +0000 UTC m=+1099.041182559" watchObservedRunningTime="2026-10-30 18:15:36.346378869 +0000 UTC m=+1099.052564226" Oct 30 18:15:41.293001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.292970 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:41.524315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.524260 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:41.778281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.778245 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:41.850781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.850749 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:15:41.855607 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.855580 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.885747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.885722 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:15:41.891344 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891322 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.891471 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891351 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.891471 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891384 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cc4m\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.891471 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891444 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.891624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891600 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.891658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891648 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.891708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.891697 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.920395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.920364 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:15:41.924165 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.924139 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.940750 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.940718 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:15:41.940909 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.940891 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-vgnd5\"" Oct 30 18:15:41.940994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.940980 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:15:41.941068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941056 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:15:41.941167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941136 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:15:41.941284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941269 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:15:41.941375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941362 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:15:41.941528 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941512 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-ls454\"" Oct 30 18:15:41.941616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941603 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:15:41.941692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941672 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:15:41.941809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941747 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:15:41.941864 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.941817 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:15:41.951608 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.951576 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:15:41.992976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.992947 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.992981 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.993097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993001 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.993097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993019 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993154 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993171 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993194 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cc4m\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.993247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993213 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.993247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993242 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5m7j\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993421 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.993540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993453 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.993540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993523 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993562 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993587 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993780 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993843 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.993904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993891 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.994016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993920 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.994016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993942 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.994016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.993981 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:41.995593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.995563 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.995744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.995728 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.995816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.995776 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.995851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.995825 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.996257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.996237 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:15:41.996257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.996252 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:15:41.996257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.996241 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:15:41.996483 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.996450 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:41.998106 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:41.998075 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:15:42.015778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.015760 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cc4m\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:42.095248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095223 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095267 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095321 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095441 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095358 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095441 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095386 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095441 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095414 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095495 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5m7j\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095539 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095573 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095596 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095631 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.095904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.095667 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.096418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.096075 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.096418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.096331 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.098208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.098177 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.098485 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.098437 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.098567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.098488 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.098567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.098494 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.098780 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.098752 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.099040 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.099019 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.099078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.099041 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.099278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.099258 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.099311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.099289 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.120672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.120653 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5m7j\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:42.176331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.176307 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:42.507790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:42.507697 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:15:42.512719 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:15:42.512693 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba07d3e9_2718_4195_977d_131737891144.slice/crio-544be53c35b7f6a97bf265f4e9b35b0a1d59302ac194b73e7d2326c62c7124e8 WatchSource:0}: Error finding container 544be53c35b7f6a97bf265f4e9b35b0a1d59302ac194b73e7d2326c62c7124e8: Status 404 returned error can't find the container with id 544be53c35b7f6a97bf265f4e9b35b0a1d59302ac194b73e7d2326c62c7124e8 Oct 30 18:15:43.332916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:43.332877 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerStarted","Data":"79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a"} Oct 30 18:15:43.333097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:43.332953 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerStarted","Data":"544be53c35b7f6a97bf265f4e9b35b0a1d59302ac194b73e7d2326c62c7124e8"} Oct 30 18:15:45.289260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:45.289232 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:15:47.199214 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:15:47.199179 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:15:47.356030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:47.355992 6966 generic.go:356] "Generic (PLEG): container finished" podID="ba07d3e9-2718-4195-977d-131737891144" containerID="79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a" exitCode=0 Oct 30 18:15:47.356201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:47.356080 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerDied","Data":"79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a"} Oct 30 18:15:48.363799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:48.363769 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerStarted","Data":"ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de"} Oct 30 18:15:48.364226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:48.363828 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerStarted","Data":"d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e"} Oct 30 18:15:48.364226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:48.364007 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:48.433841 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:48.433792 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.749663183 podStartE2EDuration="7.433777865s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684114682s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:48.421362531 +0000 UTC m=+1111.127547888" watchObservedRunningTime="2026-10-30 18:15:48.433777865 +0000 UTC m=+1111.139963223" Oct 30 18:15:49.156450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.156417 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.183236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.183209 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.257830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.257799 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.257975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.257915 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.271537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.271509 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") DevicePath \"csi-83df9dfad9fd2ac35dca1441af096f35871cdbdee4927aba3c5e3091d60e99b5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.373059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.373034 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.398632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.398606 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.447272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.447211 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:15:49.778762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:49.778687 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:15:49.783093 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:15:49.783065 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7047b0d_c1f3_4d8b_9009_92ff228a86f1.slice/crio-057e191a8287c00733020144447fc2c3711d7d232b6b13aa13f6160fdeb2ef37 WatchSource:0}: Error finding container 057e191a8287c00733020144447fc2c3711d7d232b6b13aa13f6160fdeb2ef37: Status 404 returned error can't find the container with id 057e191a8287c00733020144447fc2c3711d7d232b6b13aa13f6160fdeb2ef37 Oct 30 18:15:50.377155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:50.377122 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerStarted","Data":"d932f41232be0464d4908e0fee90e0a82a8e3f931fba1c0f1a66e0cfbc2fa0d9"} Oct 30 18:15:50.377540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:50.377195 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerStarted","Data":"057e191a8287c00733020144447fc2c3711d7d232b6b13aa13f6160fdeb2ef37"} Oct 30 18:15:54.371591 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:54.371560 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:15:54.400356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:54.400327 6966 generic.go:356] "Generic (PLEG): container finished" podID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerID="d932f41232be0464d4908e0fee90e0a82a8e3f931fba1c0f1a66e0cfbc2fa0d9" exitCode=0 Oct 30 18:15:54.400515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:54.400366 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerDied","Data":"d932f41232be0464d4908e0fee90e0a82a8e3f931fba1c0f1a66e0cfbc2fa0d9"} Oct 30 18:15:55.411416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:55.411385 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerStarted","Data":"ec9d260e94e1c8e9da0b336c84f697ebda42942ce2b5cccb2176663f371d2984"} Oct 30 18:15:55.411806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:55.411445 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerStarted","Data":"50a2e119d7a5ad8c3ffcdf93d4c603c9c7e159dd614538f2f3daae2c3c0ffeee"} Oct 30 18:15:55.411806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:55.411492 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerStarted","Data":"81422238d284177f6ad5e030885061bb0732f913662ed919bc2b9f565cfd8932"} Oct 30 18:15:55.495406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:55.495356 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=11.211750982 podStartE2EDuration="14.495340098s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.283589116s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:55.483910783 +0000 UTC m=+1118.190096140" watchObservedRunningTime="2026-10-30 18:15:55.495340098 +0000 UTC m=+1118.201525488" Oct 30 18:15:59.447620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:15:59.447589 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:16:04.448368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:04.448333 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:16:04.453738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:04.453714 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:16:04.462822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:04.462805 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:16:16.696363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:16.696321 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:16.925239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:16.925204 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:17.149051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:17.149020 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:17.199128 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:16:17.199042 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:16:18.373508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:18.373477 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:16:18.373945 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:18.373686 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" containerName="otc-container" containerID="cri-o://4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602" gracePeriod=30 Oct 30 18:16:18.628519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:18.628412 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:18.852301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:18.852262 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:20.721844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.721820 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:20.836060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836036 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5\" (UniqueName: \"kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5\") pod \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " Oct 30 18:16:20.836199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836103 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs\") pod \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " Oct 30 18:16:20.836199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836146 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\") pod \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " Oct 30 18:16:20.836199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836178 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal\") pod \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " Oct 30 18:16:20.836355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836205 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert\") pod \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " Oct 30 18:16:20.836355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836238 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs\") pod \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\" (UID: \"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76\") " Oct 30 18:16:20.836784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.836752 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal" pod "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:20.838471 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.838433 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs" pod "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:20.838783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.838761 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs" pod "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:20.838853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.838822 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5" pod "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76"). InnerVolumeSpecName "kube-api-access-mbvr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:20.838923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.838903 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs" pod "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:20.838976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.838945 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert" pod "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" (UID: "ad20e6e1-68cf-4caf-9f5c-0a36309cbd76"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:20.937475 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.937439 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbvr5\" (UniqueName: \"kubernetes.io/projected/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-kube-api-access-mbvr5\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.937593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.937483 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.937593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.937499 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.937593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.937513 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.937593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.937529 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.937593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:20.937543 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:16:21.548394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.548364 6966 generic.go:356] "Generic (PLEG): container finished" podID="ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" containerID="4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602" exitCode=0 Oct 30 18:16:21.548549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.548400 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.548549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.548440 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76","Type":"ContainerDied","Data":"4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602"} Oct 30 18:16:21.548549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.548505 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"ad20e6e1-68cf-4caf-9f5c-0a36309cbd76","Type":"ContainerDied","Data":"ad860800dda8071684e5c8add92f32dcd2c8a665d976ef6779f1d8f46722d1b1"} Oct 30 18:16:21.548549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.548528 6966 scope.go:122] "RemoveContainer" containerID="4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602" Oct 30 18:16:21.557799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.557783 6966 scope.go:122] "RemoveContainer" containerID="4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602" Oct 30 18:16:21.558077 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:16:21.558060 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602\": container with ID starting with 4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602 not found: ID does not exist" containerID="4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602" Oct 30 18:16:21.558136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.558089 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602"} err="failed to get container status \"4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602\": rpc error: code = NotFound desc = could not find container \"4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602\": container with ID starting with 4385205dc80c618c0885b95051b8f3515684f9489678ad11d2c0d37095a14602 not found: ID does not exist" Oct 30 18:16:21.597134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.597108 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:16:21.608118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.608090 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:16:21.639952 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.639931 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:16:21.640376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.640363 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" containerName="otc-container" Oct 30 18:16:21.640452 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.640384 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" containerName="otc-container" Oct 30 18:16:21.640525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.640480 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" containerName="otc-container" Oct 30 18:16:21.650285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.650268 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.664682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.664589 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:16:21.664808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.664792 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:16:21.664895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.664875 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:16:21.664951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.664883 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:16:21.664951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.664909 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:16:21.665169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.665135 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:16:21.666702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.666682 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:16:21.744358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.744295 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.744358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.744330 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.744755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.744414 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.744755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.744456 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szn6z\" (UniqueName: \"kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.744755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.744541 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.744755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.744587 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.845797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.845768 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.845936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.845802 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.845936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.845840 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.845936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.845876 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szn6z\" (UniqueName: \"kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.845936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.845916 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.846141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.845950 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.846619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.846572 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.848647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.848623 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.848937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.848916 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.849000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.848969 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.849056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.848975 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.868496 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.868451 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szn6z\" (UniqueName: \"kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z\") pod \"data-science-collector-collector-1\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:21.963244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:21.963218 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:16:22.264593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:22.264521 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:16:22.270014 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:16:22.269977 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod243a28e7_b67b_4351_8943_746b8ed2947b.slice/crio-8166d6e917203761f4c4a42e57e77b85352c817e5d998386bc6b1fc6575bcc17 WatchSource:0}: Error finding container 8166d6e917203761f4c4a42e57e77b85352c817e5d998386bc6b1fc6575bcc17: Status 404 returned error can't find the container with id 8166d6e917203761f4c4a42e57e77b85352c817e5d998386bc6b1fc6575bcc17 Oct 30 18:16:22.272282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:22.272249 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:16:22.272360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:22.272310 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:16:22.556498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:22.556401 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"243a28e7-b67b-4351-8943-746b8ed2947b","Type":"ContainerStarted","Data":"f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3"} Oct 30 18:16:22.556498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:22.556450 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"243a28e7-b67b-4351-8943-746b8ed2947b","Type":"ContainerStarted","Data":"8166d6e917203761f4c4a42e57e77b85352c817e5d998386bc6b1fc6575bcc17"} Oct 30 18:16:22.598670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:22.598610 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.598593695 podStartE2EDuration="1.598593695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:22.596813542 +0000 UTC m=+1145.302998898" watchObservedRunningTime="2026-10-30 18:16:22.598593695 +0000 UTC m=+1145.304779054" Oct 30 18:16:23.411252 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:23.411215 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad20e6e1-68cf-4caf-9f5c-0a36309cbd76" path="/var/lib/kubelet/pods/ad20e6e1-68cf-4caf-9f5c-0a36309cbd76/volumes" Oct 30 18:16:40.690175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:16:40.690142 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:16:47.199252 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:16:47.199217 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:17:17.199339 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:17:17.199260 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:17:18.895799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:18.895764 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:20.526588 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:20.526552 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.199183 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:17:47.199148 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:17:47.659842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.659806 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.894290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.894258 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:47.918947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.918880 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dww26"] Oct 30 18:17:47.922592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.922570 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:47.942304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.942278 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dww26"] Oct 30 18:17:47.957809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.957785 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:47.957918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.957820 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:47.957918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:47.957851 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtsjs\" (UniqueName: \"kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.058628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.058596 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.058826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.058637 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.058826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.058763 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtsjs\" (UniqueName: \"kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.059446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.059423 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.061121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.061103 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.081384 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.081364 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtsjs\" (UniqueName: \"kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs\") pod \"korrel8r-66d6f7f696-dww26\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.237137 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.237050 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:48.532877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:48.532754 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dww26"] Oct 30 18:17:48.537096 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:17:48.537057 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dea3c54_3ba0_492b_8d7f_aabbec316041.slice/crio-8dd472363db4f13e22755ccfa6ccb79d9b988188564bc90d1c7899e6ffff2b5f WatchSource:0}: Error finding container 8dd472363db4f13e22755ccfa6ccb79d9b988188564bc90d1c7899e6ffff2b5f: Status 404 returned error can't find the container with id 8dd472363db4f13e22755ccfa6ccb79d9b988188564bc90d1c7899e6ffff2b5f Oct 30 18:17:49.008838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.008795 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" event={"ID":"6dea3c54-3ba0-492b-8d7f-aabbec316041","Type":"ContainerStarted","Data":"eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672"} Oct 30 18:17:49.009007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.008849 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" event={"ID":"6dea3c54-3ba0-492b-8d7f-aabbec316041","Type":"ContainerStarted","Data":"8dd472363db4f13e22755ccfa6ccb79d9b988188564bc90d1c7899e6ffff2b5f"} Oct 30 18:17:49.009007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.008870 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:17:49.048396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.048349 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" podStartSLOduration=2.048335712 podStartE2EDuration="2.048335712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:49.04690569 +0000 UTC m=+1231.753091049" watchObservedRunningTime="2026-10-30 18:17:49.048335712 +0000 UTC m=+1231.754521068" Oct 30 18:17:49.510758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.510727 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:17:49.511227 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.511032 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="243a28e7-b67b-4351-8943-746b8ed2947b" containerName="otc-container" containerID="cri-o://f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3" gracePeriod=30 Oct 30 18:17:49.762730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:49.762644 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:50.001418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:50.001373 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.188301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.188271 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.435651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.435623 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:17:51.858821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.858795 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:51.895121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895098 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z\" (UniqueName: \"kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z\") pod \"243a28e7-b67b-4351-8943-746b8ed2947b\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " Oct 30 18:17:51.895250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895139 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal\") pod \"243a28e7-b67b-4351-8943-746b8ed2947b\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " Oct 30 18:17:51.895250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895174 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs\") pod \"243a28e7-b67b-4351-8943-746b8ed2947b\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " Oct 30 18:17:51.895250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895215 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert\") pod \"243a28e7-b67b-4351-8943-746b8ed2947b\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " Oct 30 18:17:51.895374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895271 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs\") pod \"243a28e7-b67b-4351-8943-746b8ed2947b\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " Oct 30 18:17:51.895374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895301 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs\") pod \"243a28e7-b67b-4351-8943-746b8ed2947b\" (UID: \"243a28e7-b67b-4351-8943-746b8ed2947b\") " Oct 30 18:17:51.895662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895621 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal" pod "243a28e7-b67b-4351-8943-746b8ed2947b" (UID: "243a28e7-b67b-4351-8943-746b8ed2947b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:51.895795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.895760 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/243a28e7-b67b-4351-8943-746b8ed2947b-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:17:51.897309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.897264 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs" pod "243a28e7-b67b-4351-8943-746b8ed2947b" (UID: "243a28e7-b67b-4351-8943-746b8ed2947b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:51.897309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.897289 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs" pod "243a28e7-b67b-4351-8943-746b8ed2947b" (UID: "243a28e7-b67b-4351-8943-746b8ed2947b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:51.897534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.897358 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z" pod "243a28e7-b67b-4351-8943-746b8ed2947b" (UID: "243a28e7-b67b-4351-8943-746b8ed2947b"). InnerVolumeSpecName "kube-api-access-szn6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:51.897654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.897634 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert" pod "243a28e7-b67b-4351-8943-746b8ed2947b" (UID: "243a28e7-b67b-4351-8943-746b8ed2947b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:51.897719 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.897699 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs" pod "243a28e7-b67b-4351-8943-746b8ed2947b" (UID: "243a28e7-b67b-4351-8943-746b8ed2947b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:51.996871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.996803 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szn6z\" (UniqueName: \"kubernetes.io/projected/243a28e7-b67b-4351-8943-746b8ed2947b-kube-api-access-szn6z\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:17:51.996871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.996829 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:17:51.996871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.996839 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:17:51.996871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.996848 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:17:51.996871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:51.996858 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/243a28e7-b67b-4351-8943-746b8ed2947b-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:17:52.028191 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.028161 6966 generic.go:356] "Generic (PLEG): container finished" podID="243a28e7-b67b-4351-8943-746b8ed2947b" containerID="f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3" exitCode=0 Oct 30 18:17:52.028313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.028219 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.028313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.028217 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"243a28e7-b67b-4351-8943-746b8ed2947b","Type":"ContainerDied","Data":"f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3"} Oct 30 18:17:52.028416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.028344 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"243a28e7-b67b-4351-8943-746b8ed2947b","Type":"ContainerDied","Data":"8166d6e917203761f4c4a42e57e77b85352c817e5d998386bc6b1fc6575bcc17"} Oct 30 18:17:52.028416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.028377 6966 scope.go:122] "RemoveContainer" containerID="f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3" Oct 30 18:17:52.038901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.038885 6966 scope.go:122] "RemoveContainer" containerID="f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3" Oct 30 18:17:52.039250 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:17:52.039231 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3\": container with ID starting with f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3 not found: ID does not exist" containerID="f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3" Oct 30 18:17:52.039319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.039258 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3"} err="failed to get container status \"f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3\": rpc error: code = NotFound desc = could not find container \"f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3\": container with ID starting with f8a2699e65f433bee3cb34faae19f0830a712b20502040f67f923f4e4a3396a3 not found: ID does not exist" Oct 30 18:17:52.078372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.078343 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:17:52.083220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.083183 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:17:52.118671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.118641 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:17:52.119125 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.119110 6966 cpu_manager.go:414] "containerMap: removing container" podUID="243a28e7-b67b-4351-8943-746b8ed2947b" containerName="otc-container" Oct 30 18:17:52.119187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.119126 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="243a28e7-b67b-4351-8943-746b8ed2947b" containerName="otc-container" Oct 30 18:17:52.119225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.119211 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="243a28e7-b67b-4351-8943-746b8ed2947b" containerName="otc-container" Oct 30 18:17:52.124524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.124505 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.138369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.138345 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:17:52.138369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.138356 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:17:52.138585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.138566 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:17:52.138627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.138581 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:17:52.138627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.138566 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:17:52.138691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.138647 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:17:52.144742 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.144723 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:17:52.198506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.198456 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.198882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.198533 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.198882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.198595 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.198882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.198624 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.198882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.198643 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.198882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.198665 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds5n6\" (UniqueName: \"kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.299603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.299509 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.299603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.299554 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.299603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.299580 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.299603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.299605 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.299977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.299628 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds5n6\" (UniqueName: \"kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.299977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.299734 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.300401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.300369 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.302174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.302154 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.302296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.302176 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.302479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.302447 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.302553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.302508 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.321803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.321782 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds5n6\" (UniqueName: \"kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6\") pod \"data-science-collector-collector-1\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.438030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.437992 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:17:52.739023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.738991 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:17:52.743110 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:17:52.743080 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fa07071_6b94_45e4_9087_753681e4bee9.slice/crio-a10c129dbdd9d0cde68eebfbf15bb93a0ab01dc7eedb00249a95b1f3b858fa3c WatchSource:0}: Error finding container a10c129dbdd9d0cde68eebfbf15bb93a0ab01dc7eedb00249a95b1f3b858fa3c: Status 404 returned error can't find the container with id a10c129dbdd9d0cde68eebfbf15bb93a0ab01dc7eedb00249a95b1f3b858fa3c Oct 30 18:17:52.745629 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.745590 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:17:52.745707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:52.745667 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:17:53.036995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:53.036916 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5fa07071-6b94-45e4-9087-753681e4bee9","Type":"ContainerStarted","Data":"b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322"} Oct 30 18:17:53.036995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:53.036969 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5fa07071-6b94-45e4-9087-753681e4bee9","Type":"ContainerStarted","Data":"a10c129dbdd9d0cde68eebfbf15bb93a0ab01dc7eedb00249a95b1f3b858fa3c"} Oct 30 18:17:53.079662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:53.079616 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.079601402 podStartE2EDuration="1.079601402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:53.078693761 +0000 UTC m=+1235.784879118" watchObservedRunningTime="2026-10-30 18:17:53.079601402 +0000 UTC m=+1235.785786789" Oct 30 18:17:53.412012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:17:53.411979 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="243a28e7-b67b-4351-8943-746b8ed2947b" path="/var/lib/kubelet/pods/243a28e7-b67b-4351-8943-746b8ed2947b/volumes" Oct 30 18:18:00.017127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.017098 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:18:00.089343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.089311 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx"] Oct 30 18:18:00.089599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.089559 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" podUID="3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" containerName="korrel8r" containerID="cri-o://31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a" gracePeriod=30 Oct 30 18:18:00.333101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.333080 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:18:00.477764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.477736 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config\" (UniqueName: \"kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config\") pod \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " Oct 30 18:18:00.477939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.477794 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls\" (UniqueName: \"kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls\") pod \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " Oct 30 18:18:00.477939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.477844 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp\") pod \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\" (UID: \"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f\") " Oct 30 18:18:00.478111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.478088 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config" pod "3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" (UID: "3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:00.479797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.479769 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls" pod "3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" (UID: "3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:00.479896 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.479828 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp" pod "3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" (UID: "3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f"). InnerVolumeSpecName "kube-api-access-mpwnp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:00.578773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.578752 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:00.578773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.578772 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:00.578920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:00.578781 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpwnp\" (UniqueName: \"kubernetes.io/projected/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f-kube-api-access-mpwnp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:01.084345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.084314 6966 generic.go:356] "Generic (PLEG): container finished" podID="3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" containerID="31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a" exitCode=0 Oct 30 18:18:01.084830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.084357 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" Oct 30 18:18:01.084830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.084405 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" event={"ID":"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f","Type":"ContainerDied","Data":"31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a"} Oct 30 18:18:01.084830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.084479 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx" event={"ID":"3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f","Type":"ContainerDied","Data":"4391d53a2a484c0c7cc7619f5a3b9b4a119d0e779a23ed72ee7643b65f35c5ba"} Oct 30 18:18:01.084830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.084518 6966 scope.go:122] "RemoveContainer" containerID="31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a" Oct 30 18:18:01.093427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.093285 6966 scope.go:122] "RemoveContainer" containerID="31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a" Oct 30 18:18:01.093592 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:01.093562 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a\": container with ID starting with 31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a not found: ID does not exist" containerID="31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a" Oct 30 18:18:01.093663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.093600 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a"} err="failed to get container status \"31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a\": rpc error: code = NotFound desc = could not find container \"31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a\": container with ID starting with 31adb2f1cb2008e34c521b910ffe676450068e8efb36d0d7c6aaad28cc56385a not found: ID does not exist" Oct 30 18:18:01.132378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.132357 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx"] Oct 30 18:18:01.142628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.142595 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ssfwx"] Oct 30 18:18:01.413451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:01.413381 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" path="/var/lib/kubelet/pods/3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f/volumes" Oct 30 18:18:05.227554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.227523 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:05.455242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.455212 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:05.528140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.528038 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq"] Oct 30 18:18:05.528632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.528613 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" containerName="korrel8r" Oct 30 18:18:05.528632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.528632 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" containerName="korrel8r" Oct 30 18:18:05.528783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.528713 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c5be6c1-a1b5-45f0-8ca0-8ab4e47b274f" containerName="korrel8r" Oct 30 18:18:05.531829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.531812 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.551487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.551428 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq"] Oct 30 18:18:05.624570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.624547 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.624718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.624584 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzxdg\" (UniqueName: \"kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.624718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.624699 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.725269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.725237 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.725386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.725303 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.725386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.725340 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzxdg\" (UniqueName: \"kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.726012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.725993 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.727695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.727678 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.746313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.746287 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:05.748562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.748540 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzxdg\" (UniqueName: \"kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg\") pod \"korrel8r-5cbcc89b8f-75bhq\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:05.844425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:05.844397 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:06.140605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:06.140536 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq"] Oct 30 18:18:06.145645 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:18:06.145614 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfede1aa4_9624_40ce_80ba_8cc66143dc7f.slice/crio-d53a5f96cc7d4d8a7a8b98e91c2248a5af555af4e8a6d5404c5d565390770067 WatchSource:0}: Error finding container d53a5f96cc7d4d8a7a8b98e91c2248a5af555af4e8a6d5404c5d565390770067: Status 404 returned error can't find the container with id d53a5f96cc7d4d8a7a8b98e91c2248a5af555af4e8a6d5404c5d565390770067 Oct 30 18:18:06.850455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:06.850426 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:06.850912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:06.850690 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5fa07071-6b94-45e4-9087-753681e4bee9" containerName="otc-container" containerID="cri-o://b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322" gracePeriod=30 Oct 30 18:18:07.085908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:07.082377 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:07.120735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:07.120663 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" event={"ID":"fede1aa4-9624-40ce-80ba-8cc66143dc7f","Type":"ContainerStarted","Data":"40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381"} Oct 30 18:18:07.120735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:07.120719 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" event={"ID":"fede1aa4-9624-40ce-80ba-8cc66143dc7f","Type":"ContainerStarted","Data":"d53a5f96cc7d4d8a7a8b98e91c2248a5af555af4e8a6d5404c5d565390770067"} Oct 30 18:18:07.120931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:07.120813 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:07.160060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:07.160017 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" podStartSLOduration=2.160005979 podStartE2EDuration="2.160005979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:07.158282326 +0000 UTC m=+1249.864467684" watchObservedRunningTime="2026-10-30 18:18:07.160005979 +0000 UTC m=+1249.866191337" Oct 30 18:18:07.303325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:07.303295 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:08.034037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.034005 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:08.099640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.099619 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.127383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.127355 6966 generic.go:356] "Generic (PLEG): container finished" podID="5fa07071-6b94-45e4-9087-753681e4bee9" containerID="b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322" exitCode=0 Oct 30 18:18:08.127564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.127392 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.127564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.127433 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5fa07071-6b94-45e4-9087-753681e4bee9","Type":"ContainerDied","Data":"b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322"} Oct 30 18:18:08.127564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.127491 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5fa07071-6b94-45e4-9087-753681e4bee9","Type":"ContainerDied","Data":"a10c129dbdd9d0cde68eebfbf15bb93a0ab01dc7eedb00249a95b1f3b858fa3c"} Oct 30 18:18:08.127564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.127518 6966 scope.go:122] "RemoveContainer" containerID="b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322" Oct 30 18:18:08.136033 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.136016 6966 scope.go:122] "RemoveContainer" containerID="b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322" Oct 30 18:18:08.136285 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:08.136267 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322\": container with ID starting with b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322 not found: ID does not exist" containerID="b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322" Oct 30 18:18:08.136330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.136291 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322"} err="failed to get container status \"b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322\": rpc error: code = NotFound desc = could not find container \"b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322\": container with ID starting with b75fdbd94e90563a0fac817213a506a65c42ed86f9b50538ed39622ce4d77322 not found: ID does not exist" Oct 30 18:18:08.246811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.246745 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert\") pod \"5fa07071-6b94-45e4-9087-753681e4bee9\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " Oct 30 18:18:08.246811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.246791 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6\" (UniqueName: \"kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6\") pod \"5fa07071-6b94-45e4-9087-753681e4bee9\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " Oct 30 18:18:08.247006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.246823 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal\") pod \"5fa07071-6b94-45e4-9087-753681e4bee9\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " Oct 30 18:18:08.247006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.246845 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs\") pod \"5fa07071-6b94-45e4-9087-753681e4bee9\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " Oct 30 18:18:08.247006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.246869 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs\") pod \"5fa07071-6b94-45e4-9087-753681e4bee9\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " Oct 30 18:18:08.247006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.246915 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs\") pod \"5fa07071-6b94-45e4-9087-753681e4bee9\" (UID: \"5fa07071-6b94-45e4-9087-753681e4bee9\") " Oct 30 18:18:08.247696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.247309 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal" pod "5fa07071-6b94-45e4-9087-753681e4bee9" (UID: "5fa07071-6b94-45e4-9087-753681e4bee9"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:08.249380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.249352 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs" pod "5fa07071-6b94-45e4-9087-753681e4bee9" (UID: "5fa07071-6b94-45e4-9087-753681e4bee9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:08.249611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.249446 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert" pod "5fa07071-6b94-45e4-9087-753681e4bee9" (UID: "5fa07071-6b94-45e4-9087-753681e4bee9"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:08.249791 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.249772 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6" pod "5fa07071-6b94-45e4-9087-753681e4bee9" (UID: "5fa07071-6b94-45e4-9087-753681e4bee9"). InnerVolumeSpecName "kube-api-access-ds5n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:08.249897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.249863 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs" pod "5fa07071-6b94-45e4-9087-753681e4bee9" (UID: "5fa07071-6b94-45e4-9087-753681e4bee9"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:08.250797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.250774 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs" pod "5fa07071-6b94-45e4-9087-753681e4bee9" (UID: "5fa07071-6b94-45e4-9087-753681e4bee9"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:08.347799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.347767 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:08.347799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.347797 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ds5n6\" (UniqueName: \"kubernetes.io/projected/5fa07071-6b94-45e4-9087-753681e4bee9-kube-api-access-ds5n6\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:08.347963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.347814 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5fa07071-6b94-45e4-9087-753681e4bee9-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:08.347963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.347827 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:08.347963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.347839 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:08.347963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.347852 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fa07071-6b94-45e4-9087-753681e4bee9-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:08.471427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.471400 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:08.482429 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.482406 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:08.516508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.516435 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:08.517454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.517419 6966 cpu_manager.go:414] "containerMap: removing container" podUID="5fa07071-6b94-45e4-9087-753681e4bee9" containerName="otc-container" Oct 30 18:18:08.517454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.517445 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fa07071-6b94-45e4-9087-753681e4bee9" containerName="otc-container" Oct 30 18:18:08.517720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.517705 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fa07071-6b94-45e4-9087-753681e4bee9" containerName="otc-container" Oct 30 18:18:08.526722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.526701 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.540368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.540345 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:08.547051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.545425 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:18:08.547051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.545425 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:18:08.547051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.545790 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:18:08.547051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.545908 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:18:08.549764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.549747 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:18:08.549871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.549778 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:18:08.650374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.650348 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.650556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.650390 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.650627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.650547 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.650627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.650605 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.650713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.650626 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct488\" (UniqueName: \"kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.650713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.650662 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.751665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.751638 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.751846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.751681 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.751846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.751699 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct488\" (UniqueName: \"kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.751846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.751746 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.751846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.751796 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.752076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.751959 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.752542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.752512 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.754482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.754437 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.754726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.754710 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.754896 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.754875 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.754937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.754925 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.774118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.774074 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct488\" (UniqueName: \"kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488\") pod \"data-science-collector-collector-1\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:08.841214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:08.841196 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:09.140374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:09.140343 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:09.144965 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:18:09.144938 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ca2a39_afbe_49d7_af91_6c3ea4437bcf.slice/crio-2f4c34fab85246a7bacd290ae2546b73e17792a6d24295d063472a16035c7400 WatchSource:0}: Error finding container 2f4c34fab85246a7bacd290ae2546b73e17792a6d24295d063472a16035c7400: Status 404 returned error can't find the container with id 2f4c34fab85246a7bacd290ae2546b73e17792a6d24295d063472a16035c7400 Oct 30 18:18:09.147249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:09.147219 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:18:09.147333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:09.147278 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:18:09.411853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:09.411785 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fa07071-6b94-45e4-9087-753681e4bee9" path="/var/lib/kubelet/pods/5fa07071-6b94-45e4-9087-753681e4bee9/volumes" Oct 30 18:18:10.141451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:10.141416 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"24ca2a39-afbe-49d7-af91-6c3ea4437bcf","Type":"ContainerStarted","Data":"53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403"} Oct 30 18:18:10.141858 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:10.141492 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"24ca2a39-afbe-49d7-af91-6c3ea4437bcf","Type":"ContainerStarted","Data":"2f4c34fab85246a7bacd290ae2546b73e17792a6d24295d063472a16035c7400"} Oct 30 18:18:10.184473 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:10.184422 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.184408166 podStartE2EDuration="2.184408166s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:10.183504963 +0000 UTC m=+1252.889690321" watchObservedRunningTime="2026-10-30 18:18:10.184408166 +0000 UTC m=+1252.890593523" Oct 30 18:18:17.199445 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:17.199411 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:18:18.130076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.130051 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:18.202813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.202784 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dww26"] Oct 30 18:18:18.203209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.203033 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" podUID="6dea3c54-3ba0-492b-8d7f-aabbec316041" containerName="korrel8r" containerID="cri-o://eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672" gracePeriod=30 Oct 30 18:18:18.459368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.459341 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:18:18.647130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.647102 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs\" (UniqueName: \"kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs\") pod \"6dea3c54-3ba0-492b-8d7f-aabbec316041\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " Oct 30 18:18:18.647282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.647148 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config\" (UniqueName: \"kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config\") pod \"6dea3c54-3ba0-492b-8d7f-aabbec316041\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " Oct 30 18:18:18.647282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.647212 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls\" (UniqueName: \"kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls\") pod \"6dea3c54-3ba0-492b-8d7f-aabbec316041\" (UID: \"6dea3c54-3ba0-492b-8d7f-aabbec316041\") " Oct 30 18:18:18.647611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.647582 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config" pod "6dea3c54-3ba0-492b-8d7f-aabbec316041" (UID: "6dea3c54-3ba0-492b-8d7f-aabbec316041"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:18.649362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.649339 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs" pod "6dea3c54-3ba0-492b-8d7f-aabbec316041" (UID: "6dea3c54-3ba0-492b-8d7f-aabbec316041"). InnerVolumeSpecName "kube-api-access-dtsjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:18.649497 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.649386 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls" pod "6dea3c54-3ba0-492b-8d7f-aabbec316041" (UID: "6dea3c54-3ba0-492b-8d7f-aabbec316041"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:18.748431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.748372 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6dea3c54-3ba0-492b-8d7f-aabbec316041-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.748431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.748394 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6dea3c54-3ba0-492b-8d7f-aabbec316041-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.748431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:18.748404 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtsjs\" (UniqueName: \"kubernetes.io/projected/6dea3c54-3ba0-492b-8d7f-aabbec316041-kube-api-access-dtsjs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:19.055332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.055256 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:19.190041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.190006 6966 generic.go:356] "Generic (PLEG): container finished" podID="6dea3c54-3ba0-492b-8d7f-aabbec316041" containerID="eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672" exitCode=0 Oct 30 18:18:19.190195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.190063 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" Oct 30 18:18:19.190195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.190085 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" event={"ID":"6dea3c54-3ba0-492b-8d7f-aabbec316041","Type":"ContainerDied","Data":"eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672"} Oct 30 18:18:19.190195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.190164 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-dww26" event={"ID":"6dea3c54-3ba0-492b-8d7f-aabbec316041","Type":"ContainerDied","Data":"8dd472363db4f13e22755ccfa6ccb79d9b988188564bc90d1c7899e6ffff2b5f"} Oct 30 18:18:19.190195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.190188 6966 scope.go:122] "RemoveContainer" containerID="eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672" Oct 30 18:18:19.199683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.199667 6966 scope.go:122] "RemoveContainer" containerID="eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672" Oct 30 18:18:19.199939 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:19.199919 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672\": container with ID starting with eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672 not found: ID does not exist" containerID="eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672" Oct 30 18:18:19.199992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.199947 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672"} err="failed to get container status \"eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672\": rpc error: code = NotFound desc = could not find container \"eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672\": container with ID starting with eafc4b8c7e898f9c0e26f2c4bf2e15310166e6471bb9c8f6dec6536a894c4672 not found: ID does not exist" Oct 30 18:18:19.238356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.238328 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dww26"] Oct 30 18:18:19.248541 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.248518 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-dww26"] Oct 30 18:18:19.411710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:19.411680 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dea3c54-3ba0-492b-8d7f-aabbec316041" path="/var/lib/kubelet/pods/6dea3c54-3ba0-492b-8d7f-aabbec316041/volumes" Oct 30 18:18:20.045375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:20.045344 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:22.599829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.599743 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:22.903445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.903365 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:22.908775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.908752 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf"] Oct 30 18:18:22.909186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.909168 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6dea3c54-3ba0-492b-8d7f-aabbec316041" containerName="korrel8r" Oct 30 18:18:22.909186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.909183 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dea3c54-3ba0-492b-8d7f-aabbec316041" containerName="korrel8r" Oct 30 18:18:22.909337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.909254 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dea3c54-3ba0-492b-8d7f-aabbec316041" containerName="korrel8r" Oct 30 18:18:22.912888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.912872 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:22.934594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.934574 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf"] Oct 30 18:18:22.980682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.980657 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5pxv4\" (UniqueName: \"kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:22.980812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.980713 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:22.980872 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:22.980853 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.081695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.081661 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5pxv4\" (UniqueName: \"kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.081828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.081728 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.081828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.081798 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.082517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.082498 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.084192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.084173 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.104201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.104176 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5pxv4\" (UniqueName: \"kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4\") pod \"korrel8r-66d6f7f696-86kjf\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.145700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.145675 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:23.226256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.226207 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:23.524111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:23.524024 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf"] Oct 30 18:18:23.528709 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:18:23.528679 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffba33fe_c43c_4828_9788_1902761f6499.slice/crio-49afde646d6d08d5982f258e1e7d4f938fb0e512e1b5aefcb06546217ee3e522 WatchSource:0}: Error finding container 49afde646d6d08d5982f258e1e7d4f938fb0e512e1b5aefcb06546217ee3e522: Status 404 returned error can't find the container with id 49afde646d6d08d5982f258e1e7d4f938fb0e512e1b5aefcb06546217ee3e522 Oct 30 18:18:24.220317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.220282 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" event={"ID":"ffba33fe-c43c-4828-9788-1902761f6499","Type":"ContainerStarted","Data":"91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95"} Oct 30 18:18:24.220737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.220366 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:24.220737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.220386 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" event={"ID":"ffba33fe-c43c-4828-9788-1902761f6499","Type":"ContainerStarted","Data":"49afde646d6d08d5982f258e1e7d4f938fb0e512e1b5aefcb06546217ee3e522"} Oct 30 18:18:24.260142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.260093 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" podStartSLOduration=2.260079869 podStartE2EDuration="2.260079869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:24.258422956 +0000 UTC m=+1266.964608312" watchObservedRunningTime="2026-10-30 18:18:24.260079869 +0000 UTC m=+1266.966265225" Oct 30 18:18:24.331312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.331283 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:24.331544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.331521 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="24ca2a39-afbe-49d7-af91-6c3ea4437bcf" containerName="otc-container" containerID="cri-o://53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403" gracePeriod=30 Oct 30 18:18:24.571260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.571209 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:24.586788 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.586766 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:24.596829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.596810 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs\") pod \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " Oct 30 18:18:24.596930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.596857 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs\") pod \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " Oct 30 18:18:24.596930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.596892 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal\" (UniqueName: \"kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal\") pod \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " Oct 30 18:18:24.597005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.596954 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert\") pod \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " Oct 30 18:18:24.597037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.597011 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488\" (UniqueName: \"kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488\") pod \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " Oct 30 18:18:24.597079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.597046 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs\") pod \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\" (UID: \"24ca2a39-afbe-49d7-af91-6c3ea4437bcf\") " Oct 30 18:18:24.597311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.597281 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal" pod "24ca2a39-afbe-49d7-af91-6c3ea4437bcf" (UID: "24ca2a39-afbe-49d7-af91-6c3ea4437bcf"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:24.597418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.597395 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:24.599113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.599088 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert" pod "24ca2a39-afbe-49d7-af91-6c3ea4437bcf" (UID: "24ca2a39-afbe-49d7-af91-6c3ea4437bcf"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:24.599198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.599153 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs" pod "24ca2a39-afbe-49d7-af91-6c3ea4437bcf" (UID: "24ca2a39-afbe-49d7-af91-6c3ea4437bcf"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:24.599271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.599248 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488" pod "24ca2a39-afbe-49d7-af91-6c3ea4437bcf" (UID: "24ca2a39-afbe-49d7-af91-6c3ea4437bcf"). InnerVolumeSpecName "kube-api-access-ct488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:24.599455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.599431 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs" pod "24ca2a39-afbe-49d7-af91-6c3ea4437bcf" (UID: "24ca2a39-afbe-49d7-af91-6c3ea4437bcf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:24.599455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.599443 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs" pod "24ca2a39-afbe-49d7-af91-6c3ea4437bcf" (UID: "24ca2a39-afbe-49d7-af91-6c3ea4437bcf"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:24.697842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.697810 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:24.697842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.697837 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct488\" (UniqueName: \"kubernetes.io/projected/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-kube-api-access-ct488\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:24.697842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.697848 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:24.698044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.697857 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:24.698044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.697865 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/24ca2a39-afbe-49d7-af91-6c3ea4437bcf-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:24.827653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:24.827622 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.228190 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.228110 6966 generic.go:356] "Generic (PLEG): container finished" podID="24ca2a39-afbe-49d7-af91-6c3ea4437bcf" containerID="53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403" exitCode=0 Oct 30 18:18:25.228190 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.228158 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.228666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.228195 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"24ca2a39-afbe-49d7-af91-6c3ea4437bcf","Type":"ContainerDied","Data":"53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403"} Oct 30 18:18:25.228666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.228251 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"24ca2a39-afbe-49d7-af91-6c3ea4437bcf","Type":"ContainerDied","Data":"2f4c34fab85246a7bacd290ae2546b73e17792a6d24295d063472a16035c7400"} Oct 30 18:18:25.228666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.228279 6966 scope.go:122] "RemoveContainer" containerID="53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403" Oct 30 18:18:25.238540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.238525 6966 scope.go:122] "RemoveContainer" containerID="53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403" Oct 30 18:18:25.238846 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:25.238825 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403\": container with ID starting with 53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403 not found: ID does not exist" containerID="53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403" Oct 30 18:18:25.238914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.238855 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403"} err="failed to get container status \"53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403\": rpc error: code = NotFound desc = could not find container \"53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403\": container with ID starting with 53dbbb8286fb1ac6a7ed216b8402f85eed13d19dac289e7691373339b6679403 not found: ID does not exist" Oct 30 18:18:25.308949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.308926 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:25.319626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.319605 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:25.354179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.354152 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:25.354610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.354587 6966 cpu_manager.go:414] "containerMap: removing container" podUID="24ca2a39-afbe-49d7-af91-6c3ea4437bcf" containerName="otc-container" Oct 30 18:18:25.354610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.354602 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24ca2a39-afbe-49d7-af91-6c3ea4437bcf" containerName="otc-container" Oct 30 18:18:25.354750 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.354667 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="24ca2a39-afbe-49d7-af91-6c3ea4437bcf" containerName="otc-container" Oct 30 18:18:25.359246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.359228 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.372933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.372915 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:18:25.379914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.379894 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:25.381495 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.381451 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:18:25.384256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.384228 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:18:25.384256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.384237 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:18:25.384256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.384246 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:18:25.384432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.384292 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:18:25.403955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.403932 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.404056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.404009 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.404129 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.404103 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.404183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.404158 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.404245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.404234 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.404301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.404284 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhln\" (UniqueName: \"kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.412393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.412368 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24ca2a39-afbe-49d7-af91-6c3ea4437bcf" path="/var/lib/kubelet/pods/24ca2a39-afbe-49d7-af91-6c3ea4437bcf/volumes" Oct 30 18:18:25.505551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.505499 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.505551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.505539 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.505700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.505574 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhln\" (UniqueName: \"kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.505700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.505602 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.505700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.505642 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.505700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.505683 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.506339 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.506308 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.508123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.508101 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.508223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.508143 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.508274 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.508235 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.508341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.508322 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.528366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.528330 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhln\" (UniqueName: \"kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln\") pod \"data-science-collector-collector-1\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.673302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.673273 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:25.757908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.757836 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:25.973310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.973243 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:25.977393 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:18:25.977367 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5af69371_3074_4344_a338_5c554951b1ff.slice/crio-7fbfde183caa81b62e902f3073820bf4798c8c7eb8199a90cd4554b0296f5172 WatchSource:0}: Error finding container 7fbfde183caa81b62e902f3073820bf4798c8c7eb8199a90cd4554b0296f5172: Status 404 returned error can't find the container with id 7fbfde183caa81b62e902f3073820bf4798c8c7eb8199a90cd4554b0296f5172 Oct 30 18:18:25.980126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.980090 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:18:25.980232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.980168 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:18:25.988601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:25.988578 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:26.237215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:26.237148 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5af69371-3074-4344-a338-5c554951b1ff","Type":"ContainerStarted","Data":"621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4"} Oct 30 18:18:26.237215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:26.237200 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5af69371-3074-4344-a338-5c554951b1ff","Type":"ContainerStarted","Data":"7fbfde183caa81b62e902f3073820bf4798c8c7eb8199a90cd4554b0296f5172"} Oct 30 18:18:26.279315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:26.279221 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.2792026060000001 podStartE2EDuration="1.279202606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:26.277778636 +0000 UTC m=+1268.983963993" watchObservedRunningTime="2026-10-30 18:18:26.279202606 +0000 UTC m=+1268.985387966" Oct 30 18:18:35.231030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.230950 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:35.300280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.300247 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq"] Oct 30 18:18:35.300536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.300486 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" podUID="fede1aa4-9624-40ce-80ba-8cc66143dc7f" containerName="korrel8r" containerID="cri-o://40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381" gracePeriod=30 Oct 30 18:18:35.559596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.559569 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:35.694071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.694041 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls\" (UniqueName: \"kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls\") pod \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " Oct 30 18:18:35.694232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.694078 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config\" (UniqueName: \"kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config\") pod \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " Oct 30 18:18:35.694232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.694110 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg\" (UniqueName: \"kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg\") pod \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\" (UID: \"fede1aa4-9624-40ce-80ba-8cc66143dc7f\") " Oct 30 18:18:35.694555 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.694530 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config" pod "fede1aa4-9624-40ce-80ba-8cc66143dc7f" (UID: "fede1aa4-9624-40ce-80ba-8cc66143dc7f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:35.696226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.696202 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls" pod "fede1aa4-9624-40ce-80ba-8cc66143dc7f" (UID: "fede1aa4-9624-40ce-80ba-8cc66143dc7f"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:35.696310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.696271 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg" pod "fede1aa4-9624-40ce-80ba-8cc66143dc7f" (UID: "fede1aa4-9624-40ce-80ba-8cc66143dc7f"). InnerVolumeSpecName "kube-api-access-tzxdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:35.795321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.795253 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fede1aa4-9624-40ce-80ba-8cc66143dc7f-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:35.795321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.795278 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fede1aa4-9624-40ce-80ba-8cc66143dc7f-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:35.795321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:35.795292 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzxdg\" (UniqueName: \"kubernetes.io/projected/fede1aa4-9624-40ce-80ba-8cc66143dc7f-kube-api-access-tzxdg\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:36.297288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.297257 6966 generic.go:356] "Generic (PLEG): container finished" podID="fede1aa4-9624-40ce-80ba-8cc66143dc7f" containerID="40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381" exitCode=0 Oct 30 18:18:36.297696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.297300 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" event={"ID":"fede1aa4-9624-40ce-80ba-8cc66143dc7f","Type":"ContainerDied","Data":"40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381"} Oct 30 18:18:36.297696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.297311 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" Oct 30 18:18:36.297696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.297340 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq" event={"ID":"fede1aa4-9624-40ce-80ba-8cc66143dc7f","Type":"ContainerDied","Data":"d53a5f96cc7d4d8a7a8b98e91c2248a5af555af4e8a6d5404c5d565390770067"} Oct 30 18:18:36.297696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.297361 6966 scope.go:122] "RemoveContainer" containerID="40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381" Oct 30 18:18:36.307220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.307047 6966 scope.go:122] "RemoveContainer" containerID="40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381" Oct 30 18:18:36.307325 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:36.307308 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381\": container with ID starting with 40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381 not found: ID does not exist" containerID="40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381" Oct 30 18:18:36.307374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.307333 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381"} err="failed to get container status \"40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381\": rpc error: code = NotFound desc = could not find container \"40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381\": container with ID starting with 40599e3c2ba87d8e0dd8a5eafdf27236f0e0965f00180db4622e606ef2b08381 not found: ID does not exist" Oct 30 18:18:36.346151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.346119 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq"] Oct 30 18:18:36.356519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:36.356493 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-75bhq"] Oct 30 18:18:37.411085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:37.411047 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fede1aa4-9624-40ce-80ba-8cc66143dc7f" path="/var/lib/kubelet/pods/fede1aa4-9624-40ce-80ba-8cc66143dc7f/volumes" Oct 30 18:18:40.061661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.061625 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.364355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.364255 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.382795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.382762 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx"] Oct 30 18:18:40.383212 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.383196 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fede1aa4-9624-40ce-80ba-8cc66143dc7f" containerName="korrel8r" Oct 30 18:18:40.383212 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.383212 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fede1aa4-9624-40ce-80ba-8cc66143dc7f" containerName="korrel8r" Oct 30 18:18:40.383346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.383295 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fede1aa4-9624-40ce-80ba-8cc66143dc7f" containerName="korrel8r" Oct 30 18:18:40.387679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.387661 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.408241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.408217 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx"] Oct 30 18:18:40.428202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.428170 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.428503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.428438 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj8p7\" (UniqueName: \"kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.428608 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.428535 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.529518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.529494 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.529636 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.529531 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj8p7\" (UniqueName: \"kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.529636 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.529558 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.530360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.530343 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.531991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.531966 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.551452 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.551429 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj8p7\" (UniqueName: \"kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7\") pod \"korrel8r-5cbcc89b8f-d7dwx\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.601430 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.601400 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:40.700635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.700580 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:40.996481 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:40.996389 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx"] Oct 30 18:18:41.000651 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:18:41.000624 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604bbb5c_5ebe_49e3_bd88_a967353ea5a9.slice/crio-06889c75a94c07a72e563dc931ad4b4ce8e9c9b72cd1d5dc60e2836718bca026 WatchSource:0}: Error finding container 06889c75a94c07a72e563dc931ad4b4ce8e9c9b72cd1d5dc60e2836718bca026: Status 404 returned error can't find the container with id 06889c75a94c07a72e563dc931ad4b4ce8e9c9b72cd1d5dc60e2836718bca026 Oct 30 18:18:41.325313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.325281 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" event={"ID":"604bbb5c-5ebe-49e3-bd88-a967353ea5a9","Type":"ContainerStarted","Data":"3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3"} Oct 30 18:18:41.325827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.325339 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" event={"ID":"604bbb5c-5ebe-49e3-bd88-a967353ea5a9","Type":"ContainerStarted","Data":"06889c75a94c07a72e563dc931ad4b4ce8e9c9b72cd1d5dc60e2836718bca026"} Oct 30 18:18:41.325827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.325366 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:41.370334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.370288 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" podStartSLOduration=1.370273485 podStartE2EDuration="1.370273485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:41.369175703 +0000 UTC m=+1284.075361061" watchObservedRunningTime="2026-10-30 18:18:41.370273485 +0000 UTC m=+1284.076458841" Oct 30 18:18:41.693689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.690312 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:41.693689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.690793 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5af69371-3074-4344-a338-5c554951b1ff" containerName="otc-container" containerID="cri-o://621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4" gracePeriod=30 Oct 30 18:18:41.941317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.941278 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:41.949420 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:41.949369 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.042960 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.042929 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln\" (UniqueName: \"kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln\") pod \"5af69371-3074-4344-a338-5c554951b1ff\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " Oct 30 18:18:42.042960 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.042962 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs\") pod \"5af69371-3074-4344-a338-5c554951b1ff\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " Oct 30 18:18:42.043196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.043026 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert\") pod \"5af69371-3074-4344-a338-5c554951b1ff\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " Oct 30 18:18:42.043196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.043055 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs\") pod \"5af69371-3074-4344-a338-5c554951b1ff\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " Oct 30 18:18:42.043196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.043101 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal\") pod \"5af69371-3074-4344-a338-5c554951b1ff\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " Oct 30 18:18:42.043196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.043184 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs\") pod \"5af69371-3074-4344-a338-5c554951b1ff\" (UID: \"5af69371-3074-4344-a338-5c554951b1ff\") " Oct 30 18:18:42.043874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.043846 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal" pod "5af69371-3074-4344-a338-5c554951b1ff" (UID: "5af69371-3074-4344-a338-5c554951b1ff"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:42.045977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.045947 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs" pod "5af69371-3074-4344-a338-5c554951b1ff" (UID: "5af69371-3074-4344-a338-5c554951b1ff"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:42.046091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.045969 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln" pod "5af69371-3074-4344-a338-5c554951b1ff" (UID: "5af69371-3074-4344-a338-5c554951b1ff"). InnerVolumeSpecName "kube-api-access-9nhln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:42.046091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.046002 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs" pod "5af69371-3074-4344-a338-5c554951b1ff" (UID: "5af69371-3074-4344-a338-5c554951b1ff"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:42.046091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.046038 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs" pod "5af69371-3074-4344-a338-5c554951b1ff" (UID: "5af69371-3074-4344-a338-5c554951b1ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:42.046209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.046127 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert" pod "5af69371-3074-4344-a338-5c554951b1ff" (UID: "5af69371-3074-4344-a338-5c554951b1ff"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:42.144377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.144343 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:42.144377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.144373 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5af69371-3074-4344-a338-5c554951b1ff-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:42.144377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.144383 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:42.144609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.144393 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nhln\" (UniqueName: \"kubernetes.io/projected/5af69371-3074-4344-a338-5c554951b1ff-kube-api-access-9nhln\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:42.144609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.144403 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:42.144609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.144414 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5af69371-3074-4344-a338-5c554951b1ff-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:42.160593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.160566 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:42.335372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.335340 6966 generic.go:356] "Generic (PLEG): container finished" podID="5af69371-3074-4344-a338-5c554951b1ff" containerID="621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4" exitCode=0 Oct 30 18:18:42.335771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.335392 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.335771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.335425 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5af69371-3074-4344-a338-5c554951b1ff","Type":"ContainerDied","Data":"621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4"} Oct 30 18:18:42.335771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.335494 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5af69371-3074-4344-a338-5c554951b1ff","Type":"ContainerDied","Data":"7fbfde183caa81b62e902f3073820bf4798c8c7eb8199a90cd4554b0296f5172"} Oct 30 18:18:42.335771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.335516 6966 scope.go:122] "RemoveContainer" containerID="621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4" Oct 30 18:18:42.344503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.344486 6966 scope.go:122] "RemoveContainer" containerID="621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4" Oct 30 18:18:42.344802 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:42.344779 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4\": container with ID starting with 621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4 not found: ID does not exist" containerID="621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4" Oct 30 18:18:42.344851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.344810 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4"} err="failed to get container status \"621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4\": rpc error: code = NotFound desc = could not find container \"621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4\": container with ID starting with 621a451e2e616fa46f53df777566b2d59b0cbf2b6274c59f0ca28316ded5d1f4 not found: ID does not exist" Oct 30 18:18:42.383807 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.383785 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:42.394572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.394553 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:42.426122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.426100 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:42.426620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.426605 6966 cpu_manager.go:414] "containerMap: removing container" podUID="5af69371-3074-4344-a338-5c554951b1ff" containerName="otc-container" Oct 30 18:18:42.426710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.426623 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5af69371-3074-4344-a338-5c554951b1ff" containerName="otc-container" Oct 30 18:18:42.426766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.426712 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5af69371-3074-4344-a338-5c554951b1ff" containerName="otc-container" Oct 30 18:18:42.430878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.430860 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.444861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.444840 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:18:42.445744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.445719 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:18:42.445847 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.445779 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:18:42.446432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.446405 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.446653 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.447196 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.448212 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.448268 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.448358 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.448399 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.448444 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt74r\" (UniqueName: \"kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.454189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.448535 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.456028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.456002 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:42.549348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.549314 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.549500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.549359 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.549500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.549398 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt74r\" (UniqueName: \"kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.549500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.549454 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.549682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.549522 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.549682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.549561 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.550172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.550148 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.552241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.552217 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.552409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.552392 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.552494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.552387 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.552563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.552496 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.571275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.571253 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt74r\" (UniqueName: \"kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r\") pod \"data-science-collector-collector-1\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.744254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.744175 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:18:42.871164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:42.871133 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:43.049861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.049751 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:43.054570 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:18:43.054536 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0ca38e_493f_4385_8ec6_2fc06a01587d.slice/crio-e84c5cf441b2367fb7d3533417aaf374ae16d4ea62bf5c447e1e78a3ecb53326 WatchSource:0}: Error finding container e84c5cf441b2367fb7d3533417aaf374ae16d4ea62bf5c447e1e78a3ecb53326: Status 404 returned error can't find the container with id e84c5cf441b2367fb7d3533417aaf374ae16d4ea62bf5c447e1e78a3ecb53326 Oct 30 18:18:43.057355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.057323 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:18:43.057449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.057381 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:18:43.345542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.345498 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e0ca38e-493f-4385-8ec6-2fc06a01587d","Type":"ContainerStarted","Data":"bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4"} Oct 30 18:18:43.346032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.345589 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e0ca38e-493f-4385-8ec6-2fc06a01587d","Type":"ContainerStarted","Data":"e84c5cf441b2367fb7d3533417aaf374ae16d4ea62bf5c447e1e78a3ecb53326"} Oct 30 18:18:43.385228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.385179 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.385165614 podStartE2EDuration="1.385165614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:43.384710382 +0000 UTC m=+1286.090895742" watchObservedRunningTime="2026-10-30 18:18:43.385165614 +0000 UTC m=+1286.091350971" Oct 30 18:18:43.414820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:43.414786 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5af69371-3074-4344-a338-5c554951b1ff" path="/var/lib/kubelet/pods/5af69371-3074-4344-a338-5c554951b1ff/volumes" Oct 30 18:18:47.199411 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:47.199376 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:18:52.338432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.338402 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:18:52.405061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.405034 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf"] Oct 30 18:18:52.405270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.405243 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" podUID="ffba33fe-c43c-4828-9788-1902761f6499" containerName="korrel8r" containerID="cri-o://91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95" gracePeriod=30 Oct 30 18:18:52.659223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.659200 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:52.742625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.742600 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config\" (UniqueName: \"kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config\") pod \"ffba33fe-c43c-4828-9788-1902761f6499\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " Oct 30 18:18:52.742769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.742699 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4\" (UniqueName: \"kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4\") pod \"ffba33fe-c43c-4828-9788-1902761f6499\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " Oct 30 18:18:52.742769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.742724 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls\" (UniqueName: \"kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls\") pod \"ffba33fe-c43c-4828-9788-1902761f6499\" (UID: \"ffba33fe-c43c-4828-9788-1902761f6499\") " Oct 30 18:18:52.742992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.742968 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config" pod "ffba33fe-c43c-4828-9788-1902761f6499" (UID: "ffba33fe-c43c-4828-9788-1902761f6499"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:52.744768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.744741 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4" pod "ffba33fe-c43c-4828-9788-1902761f6499" (UID: "ffba33fe-c43c-4828-9788-1902761f6499"). InnerVolumeSpecName "kube-api-access-5pxv4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:52.744855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.744786 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls" pod "ffba33fe-c43c-4828-9788-1902761f6499" (UID: "ffba33fe-c43c-4828-9788-1902761f6499"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:52.843658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.843635 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ffba33fe-c43c-4828-9788-1902761f6499-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:52.843658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.843656 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5pxv4\" (UniqueName: \"kubernetes.io/projected/ffba33fe-c43c-4828-9788-1902761f6499-kube-api-access-5pxv4\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:52.843799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:52.843666 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ffba33fe-c43c-4828-9788-1902761f6499-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:18:53.402755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.402720 6966 generic.go:356] "Generic (PLEG): container finished" podID="ffba33fe-c43c-4828-9788-1902761f6499" containerID="91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95" exitCode=0 Oct 30 18:18:53.403217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.402767 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" Oct 30 18:18:53.403217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.402790 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" event={"ID":"ffba33fe-c43c-4828-9788-1902761f6499","Type":"ContainerDied","Data":"91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95"} Oct 30 18:18:53.403217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.402854 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf" event={"ID":"ffba33fe-c43c-4828-9788-1902761f6499","Type":"ContainerDied","Data":"49afde646d6d08d5982f258e1e7d4f938fb0e512e1b5aefcb06546217ee3e522"} Oct 30 18:18:53.403217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.402887 6966 scope.go:122] "RemoveContainer" containerID="91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95" Oct 30 18:18:53.413015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.412994 6966 scope.go:122] "RemoveContainer" containerID="91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95" Oct 30 18:18:53.413303 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:18:53.413284 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95\": container with ID starting with 91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95 not found: ID does not exist" containerID="91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95" Oct 30 18:18:53.413368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.413313 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95"} err="failed to get container status \"91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95\": rpc error: code = NotFound desc = could not find container \"91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95\": container with ID starting with 91f19eeb93eccc8a4b495a45c8499b7e61ade22478ae4a4a2335ece5ba7a4b95 not found: ID does not exist" Oct 30 18:18:53.450396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.450368 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf"] Oct 30 18:18:53.454364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:53.454344 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-86kjf"] Oct 30 18:18:55.418041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:55.413970 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffba33fe-c43c-4828-9788-1902761f6499" path="/var/lib/kubelet/pods/ffba33fe-c43c-4828-9788-1902761f6499/volumes" Oct 30 18:18:57.088982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:57.088949 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:57.316895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:57.316856 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:58.614730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:58.614678 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:18:58.615221 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:58.614899 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2e0ca38e-493f-4385-8ec6-2fc06a01587d" containerName="otc-container" containerID="cri-o://bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4" gracePeriod=30 Oct 30 18:18:58.865262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:58.865173 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:18:59.088407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:18:59.088365 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:00.964158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:00.964129 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.019956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.019066 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert\") pod \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " Oct 30 18:19:01.019956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.019153 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs\") pod \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " Oct 30 18:19:01.019956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.019202 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r\" (UniqueName: \"kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r\") pod \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " Oct 30 18:19:01.019956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.019249 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs\") pod \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " Oct 30 18:19:01.019956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.019277 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal\") pod \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " Oct 30 18:19:01.019956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.019303 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs\") pod \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\" (UID: \"2e0ca38e-493f-4385-8ec6-2fc06a01587d\") " Oct 30 18:19:01.021536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.021495 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal" pod "2e0ca38e-493f-4385-8ec6-2fc06a01587d" (UID: "2e0ca38e-493f-4385-8ec6-2fc06a01587d"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:01.030831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.030807 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs" pod "2e0ca38e-493f-4385-8ec6-2fc06a01587d" (UID: "2e0ca38e-493f-4385-8ec6-2fc06a01587d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:01.036599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.033635 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs" pod "2e0ca38e-493f-4385-8ec6-2fc06a01587d" (UID: "2e0ca38e-493f-4385-8ec6-2fc06a01587d"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:01.036599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.033763 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs" pod "2e0ca38e-493f-4385-8ec6-2fc06a01587d" (UID: "2e0ca38e-493f-4385-8ec6-2fc06a01587d"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:01.036599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.033765 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r" pod "2e0ca38e-493f-4385-8ec6-2fc06a01587d" (UID: "2e0ca38e-493f-4385-8ec6-2fc06a01587d"). InnerVolumeSpecName "kube-api-access-jt74r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:01.036599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.033829 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert" pod "2e0ca38e-493f-4385-8ec6-2fc06a01587d" (UID: "2e0ca38e-493f-4385-8ec6-2fc06a01587d"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:01.120041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.120018 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.120041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.120042 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt74r\" (UniqueName: \"kubernetes.io/projected/2e0ca38e-493f-4385-8ec6-2fc06a01587d-kube-api-access-jt74r\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.120189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.120053 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.120189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.120062 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.120189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.120070 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.120189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.120079 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e0ca38e-493f-4385-8ec6-2fc06a01587d-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:01.453327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.453300 6966 generic.go:356] "Generic (PLEG): container finished" podID="2e0ca38e-493f-4385-8ec6-2fc06a01587d" containerID="bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4" exitCode=0 Oct 30 18:19:01.453472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.453347 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.453472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.453347 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e0ca38e-493f-4385-8ec6-2fc06a01587d","Type":"ContainerDied","Data":"bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4"} Oct 30 18:19:01.453554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.453483 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e0ca38e-493f-4385-8ec6-2fc06a01587d","Type":"ContainerDied","Data":"e84c5cf441b2367fb7d3533417aaf374ae16d4ea62bf5c447e1e78a3ecb53326"} Oct 30 18:19:01.453554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.453512 6966 scope.go:122] "RemoveContainer" containerID="bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4" Oct 30 18:19:01.462224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.462198 6966 scope.go:122] "RemoveContainer" containerID="bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4" Oct 30 18:19:01.462528 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:01.462505 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4\": container with ID starting with bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4 not found: ID does not exist" containerID="bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4" Oct 30 18:19:01.462621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.462540 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4"} err="failed to get container status \"bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4\": rpc error: code = NotFound desc = could not find container \"bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4\": container with ID starting with bc97fc5122f8820607c321c0cfd044bfa9d65872f6189b10fb4a131a09fd7ac4 not found: ID does not exist" Oct 30 18:19:01.501305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.501282 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:01.512080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.512056 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:01.544063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544040 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:01.544551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544534 6966 cpu_manager.go:414] "containerMap: removing container" podUID="2e0ca38e-493f-4385-8ec6-2fc06a01587d" containerName="otc-container" Oct 30 18:19:01.544551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544551 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0ca38e-493f-4385-8ec6-2fc06a01587d" containerName="otc-container" Oct 30 18:19:01.544658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544592 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ffba33fe-c43c-4828-9788-1902761f6499" containerName="korrel8r" Oct 30 18:19:01.544658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544602 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffba33fe-c43c-4828-9788-1902761f6499" containerName="korrel8r" Oct 30 18:19:01.545781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544773 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffba33fe-c43c-4828-9788-1902761f6499" containerName="korrel8r" Oct 30 18:19:01.545781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.544797 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0ca38e-493f-4385-8ec6-2fc06a01587d" containerName="otc-container" Oct 30 18:19:01.554015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.553999 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.573675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.573611 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-770491e3\"" Oct 30 18:19:01.576337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.574800 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:01.576337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.575211 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:01.576337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.575869 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:01.576337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.576117 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:01.580298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.579568 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:01.580389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.580311 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:01.624228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.624202 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.624345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.624248 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.624417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.624346 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97vnt\" (UniqueName: \"kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.624551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.624531 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.624605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.624569 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.624730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.624713 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.725781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.725727 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.725781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.725766 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.725942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.725802 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.725942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.725827 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97vnt\" (UniqueName: \"kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.725942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.725935 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.726099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.725963 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.726705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.726680 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.728309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.728278 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.728309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.728300 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.728425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.728288 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.728553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.728537 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.747902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.747878 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97vnt\" (UniqueName: \"kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt\") pod \"data-science-collector-collector-1\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:01.866059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:01.866029 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:02.166992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:02.166962 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:02.171693 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:19:02.171667 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f593b07_bbc3_421c_94b5_ccf694f25b64.slice/crio-a5f01e831398c88d2d44e90c8eb3ec9ae76ec56a57b2746642d27c9049dc72b6 WatchSource:0}: Error finding container a5f01e831398c88d2d44e90c8eb3ec9ae76ec56a57b2746642d27c9049dc72b6: Status 404 returned error can't find the container with id a5f01e831398c88d2d44e90c8eb3ec9ae76ec56a57b2746642d27c9049dc72b6 Oct 30 18:19:02.174022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:02.173989 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:02.174119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:02.174059 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:02.461361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:02.461278 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f593b07-bbc3-421c-94b5-ccf694f25b64","Type":"ContainerStarted","Data":"3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30"} Oct 30 18:19:02.461361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:02.461339 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f593b07-bbc3-421c-94b5-ccf694f25b64","Type":"ContainerStarted","Data":"a5f01e831398c88d2d44e90c8eb3ec9ae76ec56a57b2746642d27c9049dc72b6"} Oct 30 18:19:02.502866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:02.502819 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.502805781 podStartE2EDuration="1.502805781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:02.501205771 +0000 UTC m=+1305.207391129" watchObservedRunningTime="2026-10-30 18:19:02.502805781 +0000 UTC m=+1305.208991137" Oct 30 18:19:03.411967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:03.411932 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0ca38e-493f-4385-8ec6-2fc06a01587d" path="/var/lib/kubelet/pods/2e0ca38e-493f-4385-8ec6-2fc06a01587d/volumes" Oct 30 18:19:05.294682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:05.294650 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:13.987949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:13.987914 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:14.289953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:14.289867 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:14.510640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:14.510597 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:15.588067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:15.588028 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:15.588523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:15.588231 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="3f593b07-bbc3-421c-94b5-ccf694f25b64" containerName="otc-container" containerID="cri-o://3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30" gracePeriod=30 Oct 30 18:19:15.820640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:15.820609 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:16.032686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.032596 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:16.846473 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.846438 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:16.952329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.952261 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert\") pod \"3f593b07-bbc3-421c-94b5-ccf694f25b64\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " Oct 30 18:19:16.952329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.952295 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs\") pod \"3f593b07-bbc3-421c-94b5-ccf694f25b64\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " Oct 30 18:19:16.952329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.952316 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt\" (UniqueName: \"kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt\") pod \"3f593b07-bbc3-421c-94b5-ccf694f25b64\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " Oct 30 18:19:16.952638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.952363 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs\") pod \"3f593b07-bbc3-421c-94b5-ccf694f25b64\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " Oct 30 18:19:16.952638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.952438 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal\") pod \"3f593b07-bbc3-421c-94b5-ccf694f25b64\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " Oct 30 18:19:16.952638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.952485 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs\") pod \"3f593b07-bbc3-421c-94b5-ccf694f25b64\" (UID: \"3f593b07-bbc3-421c-94b5-ccf694f25b64\") " Oct 30 18:19:16.953249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.953196 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal" pod "3f593b07-bbc3-421c-94b5-ccf694f25b64" (UID: "3f593b07-bbc3-421c-94b5-ccf694f25b64"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:16.954847 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.954823 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs" pod "3f593b07-bbc3-421c-94b5-ccf694f25b64" (UID: "3f593b07-bbc3-421c-94b5-ccf694f25b64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:16.955273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.955228 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs" pod "3f593b07-bbc3-421c-94b5-ccf694f25b64" (UID: "3f593b07-bbc3-421c-94b5-ccf694f25b64"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:16.955273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.955259 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert" pod "3f593b07-bbc3-421c-94b5-ccf694f25b64" (UID: "3f593b07-bbc3-421c-94b5-ccf694f25b64"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:16.955427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.955288 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs" pod "3f593b07-bbc3-421c-94b5-ccf694f25b64" (UID: "3f593b07-bbc3-421c-94b5-ccf694f25b64"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:16.955500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:16.955418 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt" pod "3f593b07-bbc3-421c-94b5-ccf694f25b64" (UID: "3f593b07-bbc3-421c-94b5-ccf694f25b64"). InnerVolumeSpecName "kube-api-access-97vnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:17.053233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.053205 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f593b07-bbc3-421c-94b5-ccf694f25b64-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.053233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.053228 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.053373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.053242 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.053373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.053252 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.053373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.053262 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97vnt\" (UniqueName: \"kubernetes.io/projected/3f593b07-bbc3-421c-94b5-ccf694f25b64-kube-api-access-97vnt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.053373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.053270 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f593b07-bbc3-421c-94b5-ccf694f25b64-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.202288 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:17.202252 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:19:17.541546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.541496 6966 generic.go:356] "Generic (PLEG): container finished" podID="3f593b07-bbc3-421c-94b5-ccf694f25b64" containerID="3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30" exitCode=0 Oct 30 18:19:17.541546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.541540 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f593b07-bbc3-421c-94b5-ccf694f25b64","Type":"ContainerDied","Data":"3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30"} Oct 30 18:19:17.541757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.541588 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f593b07-bbc3-421c-94b5-ccf694f25b64","Type":"ContainerDied","Data":"a5f01e831398c88d2d44e90c8eb3ec9ae76ec56a57b2746642d27c9049dc72b6"} Oct 30 18:19:17.541757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.541609 6966 scope.go:122] "RemoveContainer" containerID="3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30" Oct 30 18:19:17.541757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.541524 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.552281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.552261 6966 scope.go:122] "RemoveContainer" containerID="3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30" Oct 30 18:19:17.552558 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:17.552527 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30\": container with ID starting with 3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30 not found: ID does not exist" containerID="3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30" Oct 30 18:19:17.552625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.552568 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30"} err="failed to get container status \"3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30\": rpc error: code = NotFound desc = could not find container \"3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30\": container with ID starting with 3228ab83a0aceb0797f8fe319220e7c0a7cc3b17ba93e7ffdf4f6b048de35b30 not found: ID does not exist" Oct 30 18:19:17.590251 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.590226 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:17.601567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.601545 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:17.634193 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.634171 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:17.634646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.634631 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3f593b07-bbc3-421c-94b5-ccf694f25b64" containerName="otc-container" Oct 30 18:19:17.634646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.634645 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f593b07-bbc3-421c-94b5-ccf694f25b64" containerName="otc-container" Oct 30 18:19:17.635537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.635029 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f593b07-bbc3-421c-94b5-ccf694f25b64" containerName="otc-container" Oct 30 18:19:17.639861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.639838 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.653951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.653930 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:17.654048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.653959 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:17.654048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.653968 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:19:17.654048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.653974 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:17.654048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.654000 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:17.654213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.654108 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:17.659424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.659406 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:17.759398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.759373 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.759524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.759476 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.759579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.759529 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.759633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.759622 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7zk8\" (UniqueName: \"kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.759675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.759660 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.759779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.759759 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.860243 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.860287 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.860313 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.860355 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7zk8\" (UniqueName: \"kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.860395 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.860454 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.863447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.861412 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.864159 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.863574 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.864159 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.863866 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.864503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.864486 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.864668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.864646 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.882301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.882275 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7zk8\" (UniqueName: \"kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8\") pod \"data-science-collector-collector-1\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:17.955318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:17.955253 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:18.256611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:18.256530 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:18.264344 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:19:18.264316 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ea9e9cd_64ba_4465_bcc9_cb653878c6df.slice/crio-4902f04d1347fbec9361814bfb56a1534b2a9e19ff56ccfc780b3e05b852badd WatchSource:0}: Error finding container 4902f04d1347fbec9361814bfb56a1534b2a9e19ff56ccfc780b3e05b852badd: Status 404 returned error can't find the container with id 4902f04d1347fbec9361814bfb56a1534b2a9e19ff56ccfc780b3e05b852badd Oct 30 18:19:18.267472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:18.267424 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:18.267582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:18.267505 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:18.548538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:18.548444 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ea9e9cd-64ba-4465-bcc9-cb653878c6df","Type":"ContainerStarted","Data":"911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429"} Oct 30 18:19:18.548538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:18.548524 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ea9e9cd-64ba-4465-bcc9-cb653878c6df","Type":"ContainerStarted","Data":"4902f04d1347fbec9361814bfb56a1534b2a9e19ff56ccfc780b3e05b852badd"} Oct 30 18:19:18.589906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:18.589857 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.589844115 podStartE2EDuration="1.589844115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:18.588719799 +0000 UTC m=+1321.294905153" watchObservedRunningTime="2026-10-30 18:19:18.589844115 +0000 UTC m=+1321.296029471" Oct 30 18:19:19.128529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:19.128491 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:19.411930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:19.411863 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f593b07-bbc3-421c-94b5-ccf694f25b64" path="/var/lib/kubelet/pods/3f593b07-bbc3-421c-94b5-ccf694f25b64/volumes" Oct 30 18:19:20.169912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:20.169881 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.792119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.792081 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:31.936736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.936702 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9"] Oct 30 18:19:31.940664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.940645 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:31.960051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.960030 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9"] Oct 30 18:19:31.985038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.985010 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9zx\" (UniqueName: \"kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:31.985141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.985051 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:31.985141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:31.985079 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.031916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.031889 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:32.086108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.086085 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9zx\" (UniqueName: \"kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.086200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.086123 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.086200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.086143 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.086922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.086904 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.088545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.088527 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.107989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.107970 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9zx\" (UniqueName: \"kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx\") pod \"korrel8r-66d6f7f696-jthj9\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.253931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.253902 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.551938 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.551867 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9"] Oct 30 18:19:32.557191 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:19:32.557165 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3879d940_be60_45ee_9dbb_889e8b29f007.slice/crio-90530aaa20df73282ef0b5ca5710919189427b05e99d476b7642a564212efbac WatchSource:0}: Error finding container 90530aaa20df73282ef0b5ca5710919189427b05e99d476b7642a564212efbac: Status 404 returned error can't find the container with id 90530aaa20df73282ef0b5ca5710919189427b05e99d476b7642a564212efbac Oct 30 18:19:32.625727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.625702 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" event={"ID":"3879d940-be60-45ee-9dbb-889e8b29f007","Type":"ContainerStarted","Data":"103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410"} Oct 30 18:19:32.625856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.625758 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" event={"ID":"3879d940-be60-45ee-9dbb-889e8b29f007","Type":"ContainerStarted","Data":"90530aaa20df73282ef0b5ca5710919189427b05e99d476b7642a564212efbac"} Oct 30 18:19:32.625856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.625789 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:32.664265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:32.664222 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" podStartSLOduration=1.664208353 podStartE2EDuration="1.664208353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:32.663732106 +0000 UTC m=+1335.369917464" watchObservedRunningTime="2026-10-30 18:19:32.664208353 +0000 UTC m=+1335.370393710" Oct 30 18:19:33.363126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:33.363100 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:33.363604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:33.363271 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5ea9e9cd-64ba-4465-bcc9-cb653878c6df" containerName="otc-container" containerID="cri-o://911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429" gracePeriod=30 Oct 30 18:19:33.706296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:33.706213 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:33.989193 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:33.989112 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.616070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.616049 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:34.637094 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.637068 6966 generic.go:356] "Generic (PLEG): container finished" podID="5ea9e9cd-64ba-4465-bcc9-cb653878c6df" containerID="911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429" exitCode=0 Oct 30 18:19:34.637204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.637113 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:34.637204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.637153 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ea9e9cd-64ba-4465-bcc9-cb653878c6df","Type":"ContainerDied","Data":"911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429"} Oct 30 18:19:34.637290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.637204 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5ea9e9cd-64ba-4465-bcc9-cb653878c6df","Type":"ContainerDied","Data":"4902f04d1347fbec9361814bfb56a1534b2a9e19ff56ccfc780b3e05b852badd"} Oct 30 18:19:34.637290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.637229 6966 scope.go:122] "RemoveContainer" containerID="911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429" Oct 30 18:19:34.646198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.646183 6966 scope.go:122] "RemoveContainer" containerID="911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429" Oct 30 18:19:34.646528 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:34.646455 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429\": container with ID starting with 911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429 not found: ID does not exist" containerID="911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429" Oct 30 18:19:34.646605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.646537 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429"} err="failed to get container status \"911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429\": rpc error: code = NotFound desc = could not find container \"911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429\": container with ID starting with 911541784caea1193e344d83782da1ac0422bca4f01de6f4c8680da5b54d8429 not found: ID does not exist" Oct 30 18:19:34.661432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.661404 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.710383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710330 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs\") pod \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " Oct 30 18:19:34.710383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710370 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal\") pod \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " Oct 30 18:19:34.710617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710399 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs\") pod \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " Oct 30 18:19:34.710617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710449 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8\" (UniqueName: \"kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8\") pod \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " Oct 30 18:19:34.710617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710526 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs\") pod \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " Oct 30 18:19:34.710617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710546 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert\") pod \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\" (UID: \"5ea9e9cd-64ba-4465-bcc9-cb653878c6df\") " Oct 30 18:19:34.711017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.710984 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal" pod "5ea9e9cd-64ba-4465-bcc9-cb653878c6df" (UID: "5ea9e9cd-64ba-4465-bcc9-cb653878c6df"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:34.712604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.712581 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs" pod "5ea9e9cd-64ba-4465-bcc9-cb653878c6df" (UID: "5ea9e9cd-64ba-4465-bcc9-cb653878c6df"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:34.712703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.712679 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs" pod "5ea9e9cd-64ba-4465-bcc9-cb653878c6df" (UID: "5ea9e9cd-64ba-4465-bcc9-cb653878c6df"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:34.712811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.712783 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert" pod "5ea9e9cd-64ba-4465-bcc9-cb653878c6df" (UID: "5ea9e9cd-64ba-4465-bcc9-cb653878c6df"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:34.713042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.713021 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8" pod "5ea9e9cd-64ba-4465-bcc9-cb653878c6df" (UID: "5ea9e9cd-64ba-4465-bcc9-cb653878c6df"). InnerVolumeSpecName "kube-api-access-b7zk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:34.713103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.713076 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs" pod "5ea9e9cd-64ba-4465-bcc9-cb653878c6df" (UID: "5ea9e9cd-64ba-4465-bcc9-cb653878c6df"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:34.811432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.811407 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7zk8\" (UniqueName: \"kubernetes.io/projected/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-kube-api-access-b7zk8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:34.811432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.811432 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:34.811605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.811443 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:34.811605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.811451 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:34.811605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.811482 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:34.811605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.811489 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5ea9e9cd-64ba-4465-bcc9-cb653878c6df-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:34.880638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.880597 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:34.980887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.980819 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:34.992056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:34.992030 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:35.027487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.027445 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:35.028372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.028121 6966 cpu_manager.go:414] "containerMap: removing container" podUID="5ea9e9cd-64ba-4465-bcc9-cb653878c6df" containerName="otc-container" Oct 30 18:19:35.028372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.028144 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ea9e9cd-64ba-4465-bcc9-cb653878c6df" containerName="otc-container" Oct 30 18:19:35.028372 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.028237 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ea9e9cd-64ba-4465-bcc9-cb653878c6df" containerName="otc-container" Oct 30 18:19:35.032977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.032954 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.047313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.047287 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:35.047451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.047337 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-50fd48e3\"" Oct 30 18:19:35.047451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.047365 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:35.047451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.047388 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:35.047451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.047425 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:35.047451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.047430 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:35.054308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.054280 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:35.113681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.113651 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.113843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.113709 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.113843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.113786 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.113843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.113841 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.113978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.113912 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8wgs\" (UniqueName: \"kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.113978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.113948 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.214554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.214523 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.214554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.214562 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.214756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.214601 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.214756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.214617 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8wgs\" (UniqueName: \"kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.214756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.214639 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.214756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.214689 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.215416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.215390 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.217319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.217294 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.217498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.217455 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.217597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.217506 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.217675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.217661 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.237001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.236953 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8wgs\" (UniqueName: \"kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs\") pod \"data-science-collector-collector-1\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.347350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.347323 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:35.418153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.418124 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ea9e9cd-64ba-4465-bcc9-cb653878c6df" path="/var/lib/kubelet/pods/5ea9e9cd-64ba-4465-bcc9-cb653878c6df/volumes" Oct 30 18:19:35.650155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.650121 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:35.654582 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:19:35.654554 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bc027a9_603e_4bb9_8b9f_571a7353cf79.slice/crio-60e840c465c14db86db6bc51a8f1c7436ec670f1bdd857f87a6088400de90c99 WatchSource:0}: Error finding container 60e840c465c14db86db6bc51a8f1c7436ec670f1bdd857f87a6088400de90c99: Status 404 returned error can't find the container with id 60e840c465c14db86db6bc51a8f1c7436ec670f1bdd857f87a6088400de90c99 Oct 30 18:19:35.657300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.657266 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:35.657375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:35.657331 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:36.652173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:36.652136 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7bc027a9-603e-4bb9-8b9f-571a7353cf79","Type":"ContainerStarted","Data":"e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4"} Oct 30 18:19:36.652669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:36.652190 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7bc027a9-603e-4bb9-8b9f-571a7353cf79","Type":"ContainerStarted","Data":"60e840c465c14db86db6bc51a8f1c7436ec670f1bdd857f87a6088400de90c99"} Oct 30 18:19:41.225374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:41.225339 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:41.460720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:41.460690 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:43.632903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.632876 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:19:43.672543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.672496 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=8.672482532 podStartE2EDuration="8.672482532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:36.708052163 +0000 UTC m=+1339.414237520" watchObservedRunningTime="2026-10-30 18:19:43.672482532 +0000 UTC m=+1346.378667886" Oct 30 18:19:43.701615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.701580 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx"] Oct 30 18:19:43.701800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.701772 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" podUID="604bbb5c-5ebe-49e3-bd88-a967353ea5a9" containerName="korrel8r" containerID="cri-o://3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3" gracePeriod=30 Oct 30 18:19:43.953362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.953341 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:19:43.987038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.987011 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls\" (UniqueName: \"kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls\") pod \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " Oct 30 18:19:43.987174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.987158 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config\" (UniqueName: \"kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config\") pod \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " Oct 30 18:19:43.987242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.987219 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7\" (UniqueName: \"kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7\") pod \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\" (UID: \"604bbb5c-5ebe-49e3-bd88-a967353ea5a9\") " Oct 30 18:19:43.987540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.987511 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config" pod "604bbb5c-5ebe-49e3-bd88-a967353ea5a9" (UID: "604bbb5c-5ebe-49e3-bd88-a967353ea5a9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:43.988946 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.988925 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls" pod "604bbb5c-5ebe-49e3-bd88-a967353ea5a9" (UID: "604bbb5c-5ebe-49e3-bd88-a967353ea5a9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:43.989085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:43.989062 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7" pod "604bbb5c-5ebe-49e3-bd88-a967353ea5a9" (UID: "604bbb5c-5ebe-49e3-bd88-a967353ea5a9"). InnerVolumeSpecName "kube-api-access-vj8p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:44.088007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.087980 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj8p7\" (UniqueName: \"kubernetes.io/projected/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-kube-api-access-vj8p7\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:44.088007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.088003 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:44.088007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.088012 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/604bbb5c-5ebe-49e3-bd88-a967353ea5a9-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:44.707269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.707235 6966 generic.go:356] "Generic (PLEG): container finished" podID="604bbb5c-5ebe-49e3-bd88-a967353ea5a9" containerID="3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3" exitCode=0 Oct 30 18:19:44.707722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.707278 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" Oct 30 18:19:44.707722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.707313 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" event={"ID":"604bbb5c-5ebe-49e3-bd88-a967353ea5a9","Type":"ContainerDied","Data":"3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3"} Oct 30 18:19:44.707722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.707365 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx" event={"ID":"604bbb5c-5ebe-49e3-bd88-a967353ea5a9","Type":"ContainerDied","Data":"06889c75a94c07a72e563dc931ad4b4ce8e9c9b72cd1d5dc60e2836718bca026"} Oct 30 18:19:44.707722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.707389 6966 scope.go:122] "RemoveContainer" containerID="3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3" Oct 30 18:19:44.716878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.716860 6966 scope.go:122] "RemoveContainer" containerID="3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3" Oct 30 18:19:44.717162 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:44.717145 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3\": container with ID starting with 3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3 not found: ID does not exist" containerID="3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3" Oct 30 18:19:44.717208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.717170 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3"} err="failed to get container status \"3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3\": rpc error: code = NotFound desc = could not find container \"3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3\": container with ID starting with 3d019a248d6e2973d366bb300cef17d6e6a4908bb19ff7dbb6518a524d99afd3 not found: ID does not exist" Oct 30 18:19:44.757145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.757124 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx"] Oct 30 18:19:44.768321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:44.768297 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-d7dwx"] Oct 30 18:19:45.411820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:45.411788 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604bbb5c-5ebe-49e3-bd88-a967353ea5a9" path="/var/lib/kubelet/pods/604bbb5c-5ebe-49e3-bd88-a967353ea5a9/volumes" Oct 30 18:19:47.200982 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:47.200942 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:19:49.521895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.521865 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4"] Oct 30 18:19:49.522403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.522306 6966 cpu_manager.go:414] "containerMap: removing container" podUID="604bbb5c-5ebe-49e3-bd88-a967353ea5a9" containerName="korrel8r" Oct 30 18:19:49.522403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.522318 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604bbb5c-5ebe-49e3-bd88-a967353ea5a9" containerName="korrel8r" Oct 30 18:19:49.522403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.522380 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="604bbb5c-5ebe-49e3-bd88-a967353ea5a9" containerName="korrel8r" Oct 30 18:19:49.527447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.527425 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.545792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.545766 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4"] Oct 30 18:19:49.634669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.634643 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.634829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.634675 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.634829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.634703 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnv2\" (UniqueName: \"kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.735718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.735693 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.735877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.735724 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.735877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.735856 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnv2\" (UniqueName: \"kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.736520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.736498 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.738108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.738089 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.757859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.757838 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnv2\" (UniqueName: \"kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2\") pod \"korrel8r-5cbcc89b8f-zkgv4\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:49.841072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:49.841050 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:50.139154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.139029 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4"] Oct 30 18:19:50.143575 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:19:50.143548 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b58ce80_1323_4e58_9235_4a32144cf1bf.slice/crio-435659840d6cafda37e6aee47a451af5b991ec39b0b3375e80311dc9fb630881 WatchSource:0}: Error finding container 435659840d6cafda37e6aee47a451af5b991ec39b0b3375e80311dc9fb630881: Status 404 returned error can't find the container with id 435659840d6cafda37e6aee47a451af5b991ec39b0b3375e80311dc9fb630881 Oct 30 18:19:50.749014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.748980 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" event={"ID":"6b58ce80-1323-4e58-9235-4a32144cf1bf","Type":"ContainerStarted","Data":"56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a"} Oct 30 18:19:50.749507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.749031 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" event={"ID":"6b58ce80-1323-4e58-9235-4a32144cf1bf","Type":"ContainerStarted","Data":"435659840d6cafda37e6aee47a451af5b991ec39b0b3375e80311dc9fb630881"} Oct 30 18:19:50.749507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.749069 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:19:50.788431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.788384 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" podStartSLOduration=1.7883711180000001 podStartE2EDuration="1.788371118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:50.787535814 +0000 UTC m=+1353.493721168" watchObservedRunningTime="2026-10-30 18:19:50.788371118 +0000 UTC m=+1353.494556474" Oct 30 18:19:50.829857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.829830 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:50.830262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:50.830071 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7bc027a9-603e-4bb9-8b9f-571a7353cf79" containerName="otc-container" containerID="cri-o://e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4" gracePeriod=30 Oct 30 18:19:51.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:51.649864 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:51.875249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:51.875220 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.078513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.078486 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.088770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.088746 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:19:52.157925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.157857 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs\") pod \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " Oct 30 18:19:52.157925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.157896 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal\") pod \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " Oct 30 18:19:52.158131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.157935 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert\") pod \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " Oct 30 18:19:52.158131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.158087 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs\" (UniqueName: \"kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs\") pod \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " Oct 30 18:19:52.158243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.158156 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs\") pod \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " Oct 30 18:19:52.158243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.158191 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs\") pod \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\" (UID: \"7bc027a9-603e-4bb9-8b9f-571a7353cf79\") " Oct 30 18:19:52.158357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.158323 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal" pod "7bc027a9-603e-4bb9-8b9f-571a7353cf79" (UID: "7bc027a9-603e-4bb9-8b9f-571a7353cf79"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:52.158615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.158589 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.160427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.160403 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs" pod "7bc027a9-603e-4bb9-8b9f-571a7353cf79" (UID: "7bc027a9-603e-4bb9-8b9f-571a7353cf79"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:52.160683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.160660 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs" pod "7bc027a9-603e-4bb9-8b9f-571a7353cf79" (UID: "7bc027a9-603e-4bb9-8b9f-571a7353cf79"). InnerVolumeSpecName "kube-api-access-g8wgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:52.160765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.160680 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs" pod "7bc027a9-603e-4bb9-8b9f-571a7353cf79" (UID: "7bc027a9-603e-4bb9-8b9f-571a7353cf79"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:52.160810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.160764 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert" pod "7bc027a9-603e-4bb9-8b9f-571a7353cf79" (UID: "7bc027a9-603e-4bb9-8b9f-571a7353cf79"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:52.160810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.160786 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs" pod "7bc027a9-603e-4bb9-8b9f-571a7353cf79" (UID: "7bc027a9-603e-4bb9-8b9f-571a7353cf79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:52.259685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.259653 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.259685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.259686 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.259885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.259703 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8wgs\" (UniqueName: \"kubernetes.io/projected/7bc027a9-603e-4bb9-8b9f-571a7353cf79-kube-api-access-g8wgs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.259885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.259719 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.259885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.259735 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7bc027a9-603e-4bb9-8b9f-571a7353cf79-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.761511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.761476 6966 generic.go:356] "Generic (PLEG): container finished" podID="7bc027a9-603e-4bb9-8b9f-571a7353cf79" containerID="e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4" exitCode=0 Oct 30 18:19:52.761688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.761520 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.761688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.761553 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7bc027a9-603e-4bb9-8b9f-571a7353cf79","Type":"ContainerDied","Data":"e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4"} Oct 30 18:19:52.761688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.761608 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7bc027a9-603e-4bb9-8b9f-571a7353cf79","Type":"ContainerDied","Data":"60e840c465c14db86db6bc51a8f1c7436ec670f1bdd857f87a6088400de90c99"} Oct 30 18:19:52.761688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.761630 6966 scope.go:122] "RemoveContainer" containerID="e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4" Oct 30 18:19:52.771438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.771415 6966 scope.go:122] "RemoveContainer" containerID="e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4" Oct 30 18:19:52.771843 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:19:52.771822 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4\": container with ID starting with e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4 not found: ID does not exist" containerID="e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4" Oct 30 18:19:52.771913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.771851 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4"} err="failed to get container status \"e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4\": rpc error: code = NotFound desc = could not find container \"e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4\": container with ID starting with e21babd5b3a951014f8a909ae858126e94e0222f6111cb31504db3bc7688cde4 not found: ID does not exist" Oct 30 18:19:52.810644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.810621 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:52.822054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.822033 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:52.853396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.853356 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:52.853901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.853882 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7bc027a9-603e-4bb9-8b9f-571a7353cf79" containerName="otc-container" Oct 30 18:19:52.853901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.853900 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bc027a9-603e-4bb9-8b9f-571a7353cf79" containerName="otc-container" Oct 30 18:19:52.854108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.854041 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bc027a9-603e-4bb9-8b9f-571a7353cf79" containerName="otc-container" Oct 30 18:19:52.864599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.858944 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.873618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.873599 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:19:52.873618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.873613 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:19:52.873744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.873599 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:19:52.873744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.873602 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:19:52.873744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.873599 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:19:52.878445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.878423 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:52.884296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.884273 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:19:52.967203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.967172 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s8br\" (UniqueName: \"kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.967365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.967267 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.967365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.967296 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.967365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.967320 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.967365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.967344 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:52.967553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:52.967391 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.068400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.068365 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.068606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.068426 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s8br\" (UniqueName: \"kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.068606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.068455 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.068606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.068522 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.068606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.068554 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.068606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.068578 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.069349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.069323 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.071586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.071559 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.071701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.071593 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.071701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.071601 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.071773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.071710 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.090781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.090760 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s8br\" (UniqueName: \"kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br\") pod \"data-science-collector-collector-1\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.175947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.175924 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:19:53.411535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.411449 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bc027a9-603e-4bb9-8b9f-571a7353cf79" path="/var/lib/kubelet/pods/7bc027a9-603e-4bb9-8b9f-571a7353cf79/volumes" Oct 30 18:19:53.475449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.475421 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:19:53.479641 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:19:53.479615 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b84e3d_29c2_49f0_8191_9961954d1fc7.slice/crio-5cbf6b63d63dd825900fde0ab7abe9576188ed82d9e459c3cf69454371c6b48e WatchSource:0}: Error finding container 5cbf6b63d63dd825900fde0ab7abe9576188ed82d9e459c3cf69454371c6b48e: Status 404 returned error can't find the container with id 5cbf6b63d63dd825900fde0ab7abe9576188ed82d9e459c3cf69454371c6b48e Oct 30 18:19:53.481819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.481790 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:53.481899 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.481851 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:19:53.768197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.768107 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"07b84e3d-29c2-49f0-8191-9961954d1fc7","Type":"ContainerStarted","Data":"d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada"} Oct 30 18:19:53.768197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.768178 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"07b84e3d-29c2-49f0-8191-9961954d1fc7","Type":"ContainerStarted","Data":"5cbf6b63d63dd825900fde0ab7abe9576188ed82d9e459c3cf69454371c6b48e"} Oct 30 18:19:53.812075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:19:53.811890 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.811876638 podStartE2EDuration="1.811876638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:53.811186102 +0000 UTC m=+1356.517371460" watchObservedRunningTime="2026-10-30 18:19:53.811876638 +0000 UTC m=+1356.518061995" Oct 30 18:20:01.756536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:01.756451 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:20:01.824644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:01.824615 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9"] Oct 30 18:20:01.824836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:01.824814 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" podUID="3879d940-be60-45ee-9dbb-889e8b29f007" containerName="korrel8r" containerID="cri-o://103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410" gracePeriod=30 Oct 30 18:20:02.076860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.076838 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:20:02.155728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.155691 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls\" (UniqueName: \"kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls\") pod \"3879d940-be60-45ee-9dbb-889e8b29f007\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " Oct 30 18:20:02.155867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.155847 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx\" (UniqueName: \"kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx\") pod \"3879d940-be60-45ee-9dbb-889e8b29f007\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " Oct 30 18:20:02.155914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.155876 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config\" (UniqueName: \"kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config\") pod \"3879d940-be60-45ee-9dbb-889e8b29f007\" (UID: \"3879d940-be60-45ee-9dbb-889e8b29f007\") " Oct 30 18:20:02.156321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.156294 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config" pod "3879d940-be60-45ee-9dbb-889e8b29f007" (UID: "3879d940-be60-45ee-9dbb-889e8b29f007"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:02.157704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.157679 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls" pod "3879d940-be60-45ee-9dbb-889e8b29f007" (UID: "3879d940-be60-45ee-9dbb-889e8b29f007"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:02.157878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.157861 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx" pod "3879d940-be60-45ee-9dbb-889e8b29f007" (UID: "3879d940-be60-45ee-9dbb-889e8b29f007"). InnerVolumeSpecName "kube-api-access-tc9zx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:02.256936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.256911 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9zx\" (UniqueName: \"kubernetes.io/projected/3879d940-be60-45ee-9dbb-889e8b29f007-kube-api-access-tc9zx\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:02.256936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.256935 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3879d940-be60-45ee-9dbb-889e8b29f007-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:02.257080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.256945 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3879d940-be60-45ee-9dbb-889e8b29f007-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:02.830028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.829995 6966 generic.go:356] "Generic (PLEG): container finished" podID="3879d940-be60-45ee-9dbb-889e8b29f007" containerID="103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410" exitCode=0 Oct 30 18:20:02.830406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.830105 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" event={"ID":"3879d940-be60-45ee-9dbb-889e8b29f007","Type":"ContainerDied","Data":"103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410"} Oct 30 18:20:02.830406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.830163 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" event={"ID":"3879d940-be60-45ee-9dbb-889e8b29f007","Type":"ContainerDied","Data":"90530aaa20df73282ef0b5ca5710919189427b05e99d476b7642a564212efbac"} Oct 30 18:20:02.830406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.830183 6966 scope.go:122] "RemoveContainer" containerID="103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410" Oct 30 18:20:02.830406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.830290 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9" Oct 30 18:20:02.843019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.842998 6966 scope.go:122] "RemoveContainer" containerID="103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410" Oct 30 18:20:02.843309 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:20:02.843288 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410\": container with ID starting with 103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410 not found: ID does not exist" containerID="103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410" Oct 30 18:20:02.843389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.843316 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410"} err="failed to get container status \"103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410\": rpc error: code = NotFound desc = could not find container \"103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410\": container with ID starting with 103aaf4787c329e64912a0f1d949b5ba7ce79cc48e7fe9923e0529ab89acb410 not found: ID does not exist" Oct 30 18:20:02.879142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.879116 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9"] Oct 30 18:20:02.889884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:02.889861 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-jthj9"] Oct 30 18:20:03.411771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:03.411736 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3879d940-be60-45ee-9dbb-889e8b29f007" path="/var/lib/kubelet/pods/3879d940-be60-45ee-9dbb-889e8b29f007/volumes" Oct 30 18:20:06.581541 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:06.581511 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:06.803252 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:06.803219 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.152533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.152498 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:08.153014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.152710 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="07b84e3d-29c2-49f0-8191-9961954d1fc7" containerName="otc-container" containerID="cri-o://d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada" gracePeriod=30 Oct 30 18:20:08.406359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.406298 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.415885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.415858 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:08.513399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513366 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs\") pod \"07b84e3d-29c2-49f0-8191-9961954d1fc7\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " Oct 30 18:20:08.513582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513416 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs\") pod \"07b84e3d-29c2-49f0-8191-9961954d1fc7\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " Oct 30 18:20:08.513582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513481 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal\") pod \"07b84e3d-29c2-49f0-8191-9961954d1fc7\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " Oct 30 18:20:08.513582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513524 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs\") pod \"07b84e3d-29c2-49f0-8191-9961954d1fc7\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " Oct 30 18:20:08.513768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513583 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br\" (UniqueName: \"kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br\") pod \"07b84e3d-29c2-49f0-8191-9961954d1fc7\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " Oct 30 18:20:08.513768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513636 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert\") pod \"07b84e3d-29c2-49f0-8191-9961954d1fc7\" (UID: \"07b84e3d-29c2-49f0-8191-9961954d1fc7\") " Oct 30 18:20:08.513984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.513949 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal" pod "07b84e3d-29c2-49f0-8191-9961954d1fc7" (UID: "07b84e3d-29c2-49f0-8191-9961954d1fc7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:08.515739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.515715 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br" pod "07b84e3d-29c2-49f0-8191-9961954d1fc7" (UID: "07b84e3d-29c2-49f0-8191-9961954d1fc7"). InnerVolumeSpecName "kube-api-access-5s8br". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:08.515861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.515821 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert" pod "07b84e3d-29c2-49f0-8191-9961954d1fc7" (UID: "07b84e3d-29c2-49f0-8191-9961954d1fc7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:08.515861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.515836 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs" pod "07b84e3d-29c2-49f0-8191-9961954d1fc7" (UID: "07b84e3d-29c2-49f0-8191-9961954d1fc7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:08.516132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.516115 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs" pod "07b84e3d-29c2-49f0-8191-9961954d1fc7" (UID: "07b84e3d-29c2-49f0-8191-9961954d1fc7"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:08.516174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.516143 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs" pod "07b84e3d-29c2-49f0-8191-9961954d1fc7" (UID: "07b84e3d-29c2-49f0-8191-9961954d1fc7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:08.614720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.614688 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.614720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.614719 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.614720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.614729 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.614892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.614740 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/07b84e3d-29c2-49f0-8191-9961954d1fc7-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.614892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.614748 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/07b84e3d-29c2-49f0-8191-9961954d1fc7-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.614892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.614756 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s8br\" (UniqueName: \"kubernetes.io/projected/07b84e3d-29c2-49f0-8191-9961954d1fc7-kube-api-access-5s8br\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:08.659023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.658965 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:08.863686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.863654 6966 generic.go:356] "Generic (PLEG): container finished" podID="07b84e3d-29c2-49f0-8191-9961954d1fc7" containerID="d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada" exitCode=0 Oct 30 18:20:08.863822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.863713 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:08.863822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.863733 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"07b84e3d-29c2-49f0-8191-9961954d1fc7","Type":"ContainerDied","Data":"d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada"} Oct 30 18:20:08.863822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.863787 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"07b84e3d-29c2-49f0-8191-9961954d1fc7","Type":"ContainerDied","Data":"5cbf6b63d63dd825900fde0ab7abe9576188ed82d9e459c3cf69454371c6b48e"} Oct 30 18:20:08.863822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.863819 6966 scope.go:122] "RemoveContainer" containerID="d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada" Oct 30 18:20:08.873776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.873751 6966 scope.go:122] "RemoveContainer" containerID="d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada" Oct 30 18:20:08.874041 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:20:08.874021 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada\": container with ID starting with d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada not found: ID does not exist" containerID="d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada" Oct 30 18:20:08.874095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.874051 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada"} err="failed to get container status \"d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada\": rpc error: code = NotFound desc = could not find container \"d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada\": container with ID starting with d0fe069b1386857f6d395d2dc47485ff27922d63dc451ba19d212b5614b7fada not found: ID does not exist" Oct 30 18:20:08.912740 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.912692 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:08.916421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.916394 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:08.954722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.954698 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:08.955150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.955129 6966 cpu_manager.go:414] "containerMap: removing container" podUID="07b84e3d-29c2-49f0-8191-9961954d1fc7" containerName="otc-container" Oct 30 18:20:08.955150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.955145 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b84e3d-29c2-49f0-8191-9961954d1fc7" containerName="otc-container" Oct 30 18:20:08.955309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.955179 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3879d940-be60-45ee-9dbb-889e8b29f007" containerName="korrel8r" Oct 30 18:20:08.955309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.955186 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3879d940-be60-45ee-9dbb-889e8b29f007" containerName="korrel8r" Oct 30 18:20:08.955309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.955249 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3879d940-be60-45ee-9dbb-889e8b29f007" containerName="korrel8r" Oct 30 18:20:08.955309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.955264 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b84e3d-29c2-49f0-8191-9961954d1fc7" containerName="otc-container" Oct 30 18:20:08.968138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.968117 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:08.983919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:08.983895 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:09.041722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.041689 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:20:09.059895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.059862 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:20:09.060004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.059899 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:20:09.060004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.059930 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:20:09.060004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.059933 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:20:09.060004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.059932 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:20:09.119269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.119246 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.119379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.119294 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.119419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.119375 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.119491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.119454 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.119581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.119566 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.119621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.119611 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnv57\" (UniqueName: \"kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.220087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.220032 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.220087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.220069 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.220489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.220095 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnv57\" (UniqueName: \"kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.220489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.220143 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.220489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.220300 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.220489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.220364 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.221148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.221116 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.222607 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.222580 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.222758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.222736 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.222822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.222740 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.222822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.222802 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.242185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.242155 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnv57\" (UniqueName: \"kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57\") pod \"data-science-collector-collector-1\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.287062 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.287038 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:09.412456 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.412417 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b84e3d-29c2-49f0-8191-9961954d1fc7" path="/var/lib/kubelet/pods/07b84e3d-29c2-49f0-8191-9961954d1fc7/volumes" Oct 30 18:20:09.590788 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.590591 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:09.594501 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:20:09.594474 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3e3e0d3_de8d_45a6_b450_bccf5cd27c83.slice/crio-0fe1480e6349ac1a0ba6c19480c7a8d5906a1467a0cf8b1e633f25beaffbaee1 WatchSource:0}: Error finding container 0fe1480e6349ac1a0ba6c19480c7a8d5906a1467a0cf8b1e633f25beaffbaee1: Status 404 returned error can't find the container with id 0fe1480e6349ac1a0ba6c19480c7a8d5906a1467a0cf8b1e633f25beaffbaee1 Oct 30 18:20:09.596898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.596867 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:09.596976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.596928 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:09.871821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.871726 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83","Type":"ContainerStarted","Data":"478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1"} Oct 30 18:20:09.871821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.871804 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83","Type":"ContainerStarted","Data":"0fe1480e6349ac1a0ba6c19480c7a8d5906a1467a0cf8b1e633f25beaffbaee1"} Oct 30 18:20:09.912523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:09.912450 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.9124366510000002 podStartE2EDuration="1.912436651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:09.912338904 +0000 UTC m=+1372.618524262" watchObservedRunningTime="2026-10-30 18:20:09.912436651 +0000 UTC m=+1372.618622008" Oct 30 18:20:17.201429 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:20:17.201389 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:20:17.421297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:17.421266 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:17.857443 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:17.857412 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:19.255534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:19.255493 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:20.286803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:20.286760 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:23.231848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:23.231810 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:23.455269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:23.455232 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:23.746053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:23.746021 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:24.830798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:24.830767 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:24.831275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:24.831000 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" containerName="otc-container" containerID="cri-o://478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1" gracePeriod=30 Oct 30 18:20:25.061501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:25.061456 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:25.281306 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:25.281215 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:27.185720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.185696 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:27.271661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.271592 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs\") pod \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " Oct 30 18:20:27.271661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.271647 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs\") pod \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " Oct 30 18:20:27.271873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.271678 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs\") pod \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " Oct 30 18:20:27.271873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.271733 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal\") pod \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " Oct 30 18:20:27.271873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.271776 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert\") pod \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " Oct 30 18:20:27.271873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.271809 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57\" (UniqueName: \"kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57\") pod \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\" (UID: \"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83\") " Oct 30 18:20:27.272231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.272199 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal" pod "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" (UID: "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:27.274085 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.274055 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert" pod "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" (UID: "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:27.274360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.274343 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs" pod "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" (UID: "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:27.274360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.274348 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57" pod "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" (UID: "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83"). InnerVolumeSpecName "kube-api-access-xnv57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:27.274521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.274389 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs" pod "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" (UID: "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:27.274581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.274559 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs" pod "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" (UID: "d3e3e0d3-de8d-45a6-b450-bccf5cd27c83"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:27.373397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.373376 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:27.373397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.373395 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:27.373547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.373405 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:27.373547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.373413 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:27.373547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.373423 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:27.373547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.373432 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnv57\" (UniqueName: \"kubernetes.io/projected/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83-kube-api-access-xnv57\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:27.972630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.972588 6966 generic.go:356] "Generic (PLEG): container finished" podID="d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" containerID="478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1" exitCode=0 Oct 30 18:20:27.972826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.972660 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:27.972826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.972670 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83","Type":"ContainerDied","Data":"478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1"} Oct 30 18:20:27.972826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.972770 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"d3e3e0d3-de8d-45a6-b450-bccf5cd27c83","Type":"ContainerDied","Data":"0fe1480e6349ac1a0ba6c19480c7a8d5906a1467a0cf8b1e633f25beaffbaee1"} Oct 30 18:20:27.972826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.972799 6966 scope.go:122] "RemoveContainer" containerID="478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1" Oct 30 18:20:27.981729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.981708 6966 scope.go:122] "RemoveContainer" containerID="478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1" Oct 30 18:20:27.981993 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:20:27.981971 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1\": container with ID starting with 478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1 not found: ID does not exist" containerID="478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1" Oct 30 18:20:27.982043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:27.982001 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1"} err="failed to get container status \"478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1\": rpc error: code = NotFound desc = could not find container \"478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1\": container with ID starting with 478e0d086e050cde3727de4a5c654627510753e4f28d0e872dd997f5a381abd1 not found: ID does not exist" Oct 30 18:20:28.021024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.020993 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:28.032215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.032192 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:28.065744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.065721 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:28.066157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.066143 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" containerName="otc-container" Oct 30 18:20:28.066226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.066158 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" containerName="otc-container" Oct 30 18:20:28.066286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.066247 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" containerName="otc-container" Oct 30 18:20:28.070852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.070835 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.085404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.085385 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:20:28.085531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.085433 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:20:28.085531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.085503 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:20:28.085633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.085582 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:20:28.085633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.085595 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:20:28.085712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.085679 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:20:28.091212 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.091194 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:28.179426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.179397 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvf52\" (UniqueName: \"kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.179602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.179522 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.179602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.179559 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.179674 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.179649 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.179674 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.179667 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.179737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.179722 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.280986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.280928 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.280986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.280956 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.280986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.280975 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.281520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.280999 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvf52\" (UniqueName: \"kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.281520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.281052 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.281520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.281072 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.281860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.281836 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.283412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.283385 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.283529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.283480 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.283729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.283713 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.283948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.283928 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.303148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.303125 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvf52\" (UniqueName: \"kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52\") pod \"data-science-collector-collector-1\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.383859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.383838 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:28.681859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.681829 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:28.686292 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:20:28.686266 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod062e0594_d6d9_42f2_b34d_222319d14e92.slice/crio-59ccc49ddbff3bf8f53efd41d28596317ca4f1d49c65b35c9b2ca5b2442b0ebe WatchSource:0}: Error finding container 59ccc49ddbff3bf8f53efd41d28596317ca4f1d49c65b35c9b2ca5b2442b0ebe: Status 404 returned error can't find the container with id 59ccc49ddbff3bf8f53efd41d28596317ca4f1d49c65b35c9b2ca5b2442b0ebe Oct 30 18:20:28.688601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.688568 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:28.688671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.688647 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:28.979963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.979891 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"062e0594-d6d9-42f2-b34d-222319d14e92","Type":"ContainerStarted","Data":"5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b"} Oct 30 18:20:28.979963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:28.979953 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"062e0594-d6d9-42f2-b34d-222319d14e92","Type":"ContainerStarted","Data":"59ccc49ddbff3bf8f53efd41d28596317ca4f1d49c65b35c9b2ca5b2442b0ebe"} Oct 30 18:20:29.021386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:29.021334 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.021319799 podStartE2EDuration="1.021319799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:29.019602335 +0000 UTC m=+1391.725787692" watchObservedRunningTime="2026-10-30 18:20:29.021319799 +0000 UTC m=+1391.727505155" Oct 30 18:20:29.411931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:29.411896 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3e3e0d3-de8d-45a6-b450-bccf5cd27c83" path="/var/lib/kubelet/pods/d3e3e0d3-de8d-45a6-b450-bccf5cd27c83/volumes" Oct 30 18:20:40.525866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:40.525816 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.738322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:40.738291 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:40.944965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:40.944925 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.413086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:42.413050 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:42.631725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:42.631695 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:47.199563 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:20:47.199525 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:20:48.223133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:48.223076 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:48.223678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:48.223308 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="062e0594-d6d9-42f2-b34d-222319d14e92" containerName="otc-container" containerID="cri-o://5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b" gracePeriod=30 Oct 30 18:20:48.918700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:48.918667 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:50.565570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.565540 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:50.678246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678182 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52\" (UniqueName: \"kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52\") pod \"062e0594-d6d9-42f2-b34d-222319d14e92\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " Oct 30 18:20:50.678246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678235 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs\") pod \"062e0594-d6d9-42f2-b34d-222319d14e92\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " Oct 30 18:20:50.678424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678258 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert\") pod \"062e0594-d6d9-42f2-b34d-222319d14e92\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " Oct 30 18:20:50.678424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678318 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs\") pod \"062e0594-d6d9-42f2-b34d-222319d14e92\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " Oct 30 18:20:50.678424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678344 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs\") pod \"062e0594-d6d9-42f2-b34d-222319d14e92\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " Oct 30 18:20:50.678424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678380 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal\" (UniqueName: \"kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal\") pod \"062e0594-d6d9-42f2-b34d-222319d14e92\" (UID: \"062e0594-d6d9-42f2-b34d-222319d14e92\") " Oct 30 18:20:50.678897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.678851 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal" pod "062e0594-d6d9-42f2-b34d-222319d14e92" (UID: "062e0594-d6d9-42f2-b34d-222319d14e92"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:50.680544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.680517 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs" pod "062e0594-d6d9-42f2-b34d-222319d14e92" (UID: "062e0594-d6d9-42f2-b34d-222319d14e92"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:50.680670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.680611 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs" pod "062e0594-d6d9-42f2-b34d-222319d14e92" (UID: "062e0594-d6d9-42f2-b34d-222319d14e92"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:50.680736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.680700 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52" pod "062e0594-d6d9-42f2-b34d-222319d14e92" (UID: "062e0594-d6d9-42f2-b34d-222319d14e92"). InnerVolumeSpecName "kube-api-access-gvf52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:50.680854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.680837 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs" pod "062e0594-d6d9-42f2-b34d-222319d14e92" (UID: "062e0594-d6d9-42f2-b34d-222319d14e92"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:50.680925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.680911 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert" pod "062e0594-d6d9-42f2-b34d-222319d14e92" (UID: "062e0594-d6d9-42f2-b34d-222319d14e92"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:50.779239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.779211 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:50.779239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.779235 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:50.779239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.779244 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/062e0594-d6d9-42f2-b34d-222319d14e92-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:50.779415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.779253 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvf52\" (UniqueName: \"kubernetes.io/projected/062e0594-d6d9-42f2-b34d-222319d14e92-kube-api-access-gvf52\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:50.779415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.779263 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:50.779415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:50.779272 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/062e0594-d6d9-42f2-b34d-222319d14e92-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:51.096671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.096639 6966 generic.go:356] "Generic (PLEG): container finished" podID="062e0594-d6d9-42f2-b34d-222319d14e92" containerID="5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b" exitCode=0 Oct 30 18:20:51.096865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.096692 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.096865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.096728 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"062e0594-d6d9-42f2-b34d-222319d14e92","Type":"ContainerDied","Data":"5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b"} Oct 30 18:20:51.096865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.096791 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"062e0594-d6d9-42f2-b34d-222319d14e92","Type":"ContainerDied","Data":"59ccc49ddbff3bf8f53efd41d28596317ca4f1d49c65b35c9b2ca5b2442b0ebe"} Oct 30 18:20:51.096865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.096820 6966 scope.go:122] "RemoveContainer" containerID="5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b" Oct 30 18:20:51.106903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.106887 6966 scope.go:122] "RemoveContainer" containerID="5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b" Oct 30 18:20:51.107189 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:20:51.107168 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b\": container with ID starting with 5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b not found: ID does not exist" containerID="5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b" Oct 30 18:20:51.107246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.107196 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b"} err="failed to get container status \"5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b\": rpc error: code = NotFound desc = could not find container \"5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b\": container with ID starting with 5b6f807dc4c096d9e9049e796b61887f94894c36c1984ad85ef453aa4dfc938b not found: ID does not exist" Oct 30 18:20:51.146820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.146795 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:51.157953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.157932 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:51.190061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.190043 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:51.190674 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.190652 6966 cpu_manager.go:414] "containerMap: removing container" podUID="062e0594-d6d9-42f2-b34d-222319d14e92" containerName="otc-container" Oct 30 18:20:51.190774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.190675 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="062e0594-d6d9-42f2-b34d-222319d14e92" containerName="otc-container" Oct 30 18:20:51.190823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.190786 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="062e0594-d6d9-42f2-b34d-222319d14e92" containerName="otc-container" Oct 30 18:20:51.194069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.194054 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.208219 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.208194 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:20:51.208314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.208227 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:20:51.208314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.208238 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:20:51.208314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.208233 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:20:51.208314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.208289 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:20:51.208453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.208358 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:20:51.220820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.220797 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:51.282888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.282861 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.283016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.282932 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.283016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.282982 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.283114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.283048 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpld7\" (UniqueName: \"kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.283114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.283089 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.283114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.283109 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.347846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.347781 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:51.383874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.383850 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.383994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.383890 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.384066 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.384020 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.384152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.384133 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.384219 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.384164 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.384287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.384213 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpld7\" (UniqueName: \"kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.384985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.384949 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.386641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.386596 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.386787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.386762 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.386787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.386781 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.386970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.386833 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.407281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.407259 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpld7\" (UniqueName: \"kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7\") pod \"data-science-collector-collector-1\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.421545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.421524 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="062e0594-d6d9-42f2-b34d-222319d14e92" path="/var/lib/kubelet/pods/062e0594-d6d9-42f2-b34d-222319d14e92/volumes" Oct 30 18:20:51.505875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.505853 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:51.554286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.554254 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:51.813029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.813003 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:51.817328 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:20:51.817306 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90783ff_c7de_4e4c_8564_79af10022de7.slice/crio-f5d8ca0bc3aa0c83778ddf1521d16aeae4cb26956c820286a56aeb026d1c0b76 WatchSource:0}: Error finding container f5d8ca0bc3aa0c83778ddf1521d16aeae4cb26956c820286a56aeb026d1c0b76: Status 404 returned error can't find the container with id f5d8ca0bc3aa0c83778ddf1521d16aeae4cb26956c820286a56aeb026d1c0b76 Oct 30 18:20:51.819518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.819488 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:51.819582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:51.819549 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:20:52.104340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:52.104256 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b90783ff-c7de-4e4c-8564-79af10022de7","Type":"ContainerStarted","Data":"64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2"} Oct 30 18:20:52.104340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:52.104316 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b90783ff-c7de-4e4c-8564-79af10022de7","Type":"ContainerStarted","Data":"f5d8ca0bc3aa0c83778ddf1521d16aeae4cb26956c820286a56aeb026d1c0b76"} Oct 30 18:20:52.144875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:52.144820 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.144804262 podStartE2EDuration="1.144804262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:52.144171725 +0000 UTC m=+1414.850357094" watchObservedRunningTime="2026-10-30 18:20:52.144804262 +0000 UTC m=+1414.850989621" Oct 30 18:20:57.554654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:57.554619 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:57.765967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:57.765931 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:57.974091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:57.974044 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:59.191384 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.191350 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:20:59.191895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.191603 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b90783ff-c7de-4e4c-8564-79af10022de7" containerName="otc-container" containerID="cri-o://64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2" gracePeriod=30 Oct 30 18:20:59.440376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.440342 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:59.441893 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.441840 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:20:59.561906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.561865 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs\") pod \"b90783ff-c7de-4e4c-8564-79af10022de7\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " Oct 30 18:20:59.562084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.561945 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs\") pod \"b90783ff-c7de-4e4c-8564-79af10022de7\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " Oct 30 18:20:59.562084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.562007 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal\") pod \"b90783ff-c7de-4e4c-8564-79af10022de7\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " Oct 30 18:20:59.562084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.562036 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs\") pod \"b90783ff-c7de-4e4c-8564-79af10022de7\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " Oct 30 18:20:59.562269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.562087 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert\") pod \"b90783ff-c7de-4e4c-8564-79af10022de7\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " Oct 30 18:20:59.562269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.562131 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7\" (UniqueName: \"kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7\") pod \"b90783ff-c7de-4e4c-8564-79af10022de7\" (UID: \"b90783ff-c7de-4e4c-8564-79af10022de7\") " Oct 30 18:20:59.562506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.562449 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal" pod "b90783ff-c7de-4e4c-8564-79af10022de7" (UID: "b90783ff-c7de-4e4c-8564-79af10022de7"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:59.564590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.564564 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs" pod "b90783ff-c7de-4e4c-8564-79af10022de7" (UID: "b90783ff-c7de-4e4c-8564-79af10022de7"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:59.564723 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.564615 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs" pod "b90783ff-c7de-4e4c-8564-79af10022de7" (UID: "b90783ff-c7de-4e4c-8564-79af10022de7"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:59.564793 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.564769 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs" pod "b90783ff-c7de-4e4c-8564-79af10022de7" (UID: "b90783ff-c7de-4e4c-8564-79af10022de7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:59.565082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.565057 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7" pod "b90783ff-c7de-4e4c-8564-79af10022de7" (UID: "b90783ff-c7de-4e4c-8564-79af10022de7"). InnerVolumeSpecName "kube-api-access-wpld7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:59.565156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.565060 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert" pod "b90783ff-c7de-4e4c-8564-79af10022de7" (UID: "b90783ff-c7de-4e4c-8564-79af10022de7"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:59.658988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.658951 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:20:59.662910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.662891 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:59.662981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.662916 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:59.662981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.662926 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpld7\" (UniqueName: \"kubernetes.io/projected/b90783ff-c7de-4e4c-8564-79af10022de7-kube-api-access-wpld7\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:59.662981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.662935 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:59.662981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.662944 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b90783ff-c7de-4e4c-8564-79af10022de7-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:20:59.662981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:20:59.662952 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b90783ff-c7de-4e4c-8564-79af10022de7-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:21:00.146985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.146954 6966 generic.go:356] "Generic (PLEG): container finished" podID="b90783ff-c7de-4e4c-8564-79af10022de7" containerID="64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2" exitCode=0 Oct 30 18:21:00.147131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.147000 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.147131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.147046 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b90783ff-c7de-4e4c-8564-79af10022de7","Type":"ContainerDied","Data":"64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2"} Oct 30 18:21:00.147131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.147096 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b90783ff-c7de-4e4c-8564-79af10022de7","Type":"ContainerDied","Data":"f5d8ca0bc3aa0c83778ddf1521d16aeae4cb26956c820286a56aeb026d1c0b76"} Oct 30 18:21:00.147131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.147119 6966 scope.go:122] "RemoveContainer" containerID="64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2" Oct 30 18:21:00.156759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.156741 6966 scope.go:122] "RemoveContainer" containerID="64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2" Oct 30 18:21:00.157031 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:21:00.157014 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2\": container with ID starting with 64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2 not found: ID does not exist" containerID="64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2" Oct 30 18:21:00.157072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.157042 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2"} err="failed to get container status \"64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2\": rpc error: code = NotFound desc = could not find container \"64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2\": container with ID starting with 64ef55a7ed8711f67a468d69ef9c629fe2d596d27b240b263dd7afcf3abb97a2 not found: ID does not exist" Oct 30 18:21:00.194896 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.194863 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:21:00.206060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.206035 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:21:00.244018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.243991 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:21:00.244515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.244498 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b90783ff-c7de-4e4c-8564-79af10022de7" containerName="otc-container" Oct 30 18:21:00.244627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.244516 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90783ff-c7de-4e4c-8564-79af10022de7" containerName="otc-container" Oct 30 18:21:00.244689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.244671 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90783ff-c7de-4e4c-8564-79af10022de7" containerName="otc-container" Oct 30 18:21:00.249337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.249318 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.263637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.263618 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:21:00.263746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.263642 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:21:00.263746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.263708 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:21:00.263823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.263788 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:21:00.263856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.263824 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:21:00.263893 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.263854 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:21:00.269474 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.269444 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:21:00.369281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.369251 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st8nj\" (UniqueName: \"kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.369435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.369291 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.369435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.369317 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.369435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.369432 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.369644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.369513 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.369644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.369552 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.470500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.470423 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.470500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.470473 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.470660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.470521 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.470660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.470555 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.470660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.470612 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.470811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.470675 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st8nj\" (UniqueName: \"kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.471405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.471378 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.473408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.473384 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.473670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.473642 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.473670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.473655 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.473784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.473660 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.492882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.492843 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-st8nj\" (UniqueName: \"kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj\") pod \"data-science-collector-collector-1\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.561607 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.561580 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:21:00.864383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.864350 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:21:00.868965 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:21:00.868937 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26538c81_6efd_46b5_abdd_9f4205b07dc5.slice/crio-45f05e459c957490f9fe3c5f4ca7d29031f85729524b4ad049e44812522a0e8a WatchSource:0}: Error finding container 45f05e459c957490f9fe3c5f4ca7d29031f85729524b4ad049e44812522a0e8a: Status 404 returned error can't find the container with id 45f05e459c957490f9fe3c5f4ca7d29031f85729524b4ad049e44812522a0e8a Oct 30 18:21:00.871138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.871107 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:21:00.871239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:00.871164 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:21:01.158487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:01.158362 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"26538c81-6efd-46b5-abdd-9f4205b07dc5","Type":"ContainerStarted","Data":"74a705e331c4a5ebc64ae71a72da2adb50b25a428edf567788246f363f5e7e32"} Oct 30 18:21:01.158487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:01.158430 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"26538c81-6efd-46b5-abdd-9f4205b07dc5","Type":"ContainerStarted","Data":"45f05e459c957490f9fe3c5f4ca7d29031f85729524b4ad049e44812522a0e8a"} Oct 30 18:21:01.200622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:01.200564 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.200549603 podStartE2EDuration="1.200549603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:01.199029372 +0000 UTC m=+1423.905214730" watchObservedRunningTime="2026-10-30 18:21:01.200549603 +0000 UTC m=+1423.906734960" Oct 30 18:21:01.412162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:01.412084 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90783ff-c7de-4e4c-8564-79af10022de7" path="/var/lib/kubelet/pods/b90783ff-c7de-4e4c-8564-79af10022de7/volumes" Oct 30 18:21:02.062819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:02.062786 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:02.294380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:02.294350 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:02.679017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:02.678980 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:04.607455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:04.607419 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:04.812404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:04.812377 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:06.114375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:06.114348 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:06.318228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:06.318193 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:08.600883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:08.600840 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:08.811273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:08.811239 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:17.199093 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:21:17.199055 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:21:19.361685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:19.361649 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:20.409669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:21:20.409620 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:21:47.199613 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:21:47.199539 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:22:17.200412 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:22:17.200374 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:22:18.033621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:22:18.033581 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:18.239043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:22:18.239012 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:19.512651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:22:19.512595 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:20.544420 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:22:20.544384 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:33.936758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:22:33.936722 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:22:47.200119 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:22:47.200084 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:23:17.199595 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:23:17.199519 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:23:19.644364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:19.644329 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:20.806082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:20.806050 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:21.005397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:21.005363 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:26.654878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:26.654838 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:26.869695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:26.869659 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:35.077135 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:35.077084 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:35.378609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:35.378500 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:35.580453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:35.580421 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:36.673827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:36.673791 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:23:36.674202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:36.673986 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="26538c81-6efd-46b5-abdd-9f4205b07dc5" containerName="otc-container" containerID="cri-o://74a705e331c4a5ebc64ae71a72da2adb50b25a428edf567788246f363f5e7e32" gracePeriod=30 Oct 30 18:23:36.897839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:36.897810 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:37.092944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:37.092910 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.283039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:38.283008 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.491794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:38.491748 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:23:38.970867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:38.970838 6966 generic.go:356] "Generic (PLEG): container finished" podID="26538c81-6efd-46b5-abdd-9f4205b07dc5" containerID="74a705e331c4a5ebc64ae71a72da2adb50b25a428edf567788246f363f5e7e32" exitCode=0 Oct 30 18:23:38.971033 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:38.970883 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"26538c81-6efd-46b5-abdd-9f4205b07dc5","Type":"ContainerDied","Data":"74a705e331c4a5ebc64ae71a72da2adb50b25a428edf567788246f363f5e7e32"} Oct 30 18:23:39.020268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.020244 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:23:39.160564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.160485 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs\") pod \"26538c81-6efd-46b5-abdd-9f4205b07dc5\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " Oct 30 18:23:39.160710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.160583 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj\" (UniqueName: \"kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj\") pod \"26538c81-6efd-46b5-abdd-9f4205b07dc5\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " Oct 30 18:23:39.160710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.160619 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs\") pod \"26538c81-6efd-46b5-abdd-9f4205b07dc5\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " Oct 30 18:23:39.160710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.160655 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs\") pod \"26538c81-6efd-46b5-abdd-9f4205b07dc5\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " Oct 30 18:23:39.160710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.160697 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert\") pod \"26538c81-6efd-46b5-abdd-9f4205b07dc5\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " Oct 30 18:23:39.160937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.160773 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal\") pod \"26538c81-6efd-46b5-abdd-9f4205b07dc5\" (UID: \"26538c81-6efd-46b5-abdd-9f4205b07dc5\") " Oct 30 18:23:39.161385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.161359 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal" pod "26538c81-6efd-46b5-abdd-9f4205b07dc5" (UID: "26538c81-6efd-46b5-abdd-9f4205b07dc5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:39.162814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.162789 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs" pod "26538c81-6efd-46b5-abdd-9f4205b07dc5" (UID: "26538c81-6efd-46b5-abdd-9f4205b07dc5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:39.163254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.163226 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs" pod "26538c81-6efd-46b5-abdd-9f4205b07dc5" (UID: "26538c81-6efd-46b5-abdd-9f4205b07dc5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:39.163340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.163232 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj" pod "26538c81-6efd-46b5-abdd-9f4205b07dc5" (UID: "26538c81-6efd-46b5-abdd-9f4205b07dc5"). InnerVolumeSpecName "kube-api-access-st8nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:39.163340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.163317 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs" pod "26538c81-6efd-46b5-abdd-9f4205b07dc5" (UID: "26538c81-6efd-46b5-abdd-9f4205b07dc5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:39.163454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.163387 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert" pod "26538c81-6efd-46b5-abdd-9f4205b07dc5" (UID: "26538c81-6efd-46b5-abdd-9f4205b07dc5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:39.261678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.261646 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.261678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.261674 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st8nj\" (UniqueName: \"kubernetes.io/projected/26538c81-6efd-46b5-abdd-9f4205b07dc5-kube-api-access-st8nj\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.261678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.261684 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.261886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.261693 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.261886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.261704 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/26538c81-6efd-46b5-abdd-9f4205b07dc5-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.261886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.261713 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/26538c81-6efd-46b5-abdd-9f4205b07dc5-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.977616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.977589 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:23:39.978061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.977581 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"26538c81-6efd-46b5-abdd-9f4205b07dc5","Type":"ContainerDied","Data":"45f05e459c957490f9fe3c5f4ca7d29031f85729524b4ad049e44812522a0e8a"} Oct 30 18:23:39.978061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:39.977721 6966 scope.go:122] "RemoveContainer" containerID="74a705e331c4a5ebc64ae71a72da2adb50b25a428edf567788246f363f5e7e32" Oct 30 18:23:40.026607 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:40.026581 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:23:40.037621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:40.037600 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:23:41.411276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:23:41.411246 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26538c81-6efd-46b5-abdd-9f4205b07dc5" path="/var/lib/kubelet/pods/26538c81-6efd-46b5-abdd-9f4205b07dc5/volumes" Oct 30 18:23:47.199732 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:23:47.199663 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:24:17.199958 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:24:17.199924 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:24:19.754958 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:19.754923 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:20.800350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:20.800314 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.366802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:45.366708 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:45.599802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:45.599769 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:47.199733 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:24:47.199699 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:24:51.095650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:51.095608 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:51.300535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:51.300504 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:51.494943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:51.494869 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.111516 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:57.111479 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.381569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:57.381480 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.657523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:57.657427 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:24:57.871201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:24:57.871174 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:14.276026 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:14.275989 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:14.577325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:14.577294 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:14.893659 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:14.893588 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:16.099421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:16.099370 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:16.336036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:16.336004 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:17.200069 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:25:17.200034 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:25:18.632661 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.632624 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:25:18.633225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.633196 6966 cpu_manager.go:414] "containerMap: removing container" podUID="26538c81-6efd-46b5-abdd-9f4205b07dc5" containerName="otc-container" Oct 30 18:25:18.633225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.633220 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26538c81-6efd-46b5-abdd-9f4205b07dc5" containerName="otc-container" Oct 30 18:25:18.633436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.633314 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="26538c81-6efd-46b5-abdd-9f4205b07dc5" containerName="otc-container" Oct 30 18:25:18.636670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.636651 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.661573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.661548 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:25:18.661697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.661556 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:25:18.661697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.661595 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:25:18.661697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.661599 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-db539ca7\"" Oct 30 18:25:18.661697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.661644 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:25:18.661697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.661647 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:25:18.674963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.674938 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:25:18.788638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.788606 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fjkc\" (UniqueName: \"kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.788638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.788639 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.788799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.788670 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.788799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.788791 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.788867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.788850 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.788904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.788885 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890105 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890135 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890165 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fjkc\" (UniqueName: \"kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890185 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890226 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890260 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.890990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.890963 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.892781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.892757 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.892892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.892826 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.892892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.892832 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.892979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.892916 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.911817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.911791 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fjkc\" (UniqueName: \"kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc\") pod \"data-science-collector-collector-1\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:18.949620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:18.949603 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:25:19.248140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.248069 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:25:19.251654 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:25:19.251623 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefe5f17_9d89_4a89_953d_dd4c6fba9587.slice/crio-7dbd3d10ae82b0bb98716cee8e0b76171711afc87748634b298d0582dc9ce624 WatchSource:0}: Error finding container 7dbd3d10ae82b0bb98716cee8e0b76171711afc87748634b298d0582dc9ce624: Status 404 returned error can't find the container with id 7dbd3d10ae82b0bb98716cee8e0b76171711afc87748634b298d0582dc9ce624 Oct 30 18:25:19.253787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.253755 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:19.253856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.253817 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:25:19.495329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.495288 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fefe5f17-9d89-4a89-953d-dd4c6fba9587","Type":"ContainerStarted","Data":"389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235"} Oct 30 18:25:19.495512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.495356 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fefe5f17-9d89-4a89-953d-dd4c6fba9587","Type":"ContainerStarted","Data":"7dbd3d10ae82b0bb98716cee8e0b76171711afc87748634b298d0582dc9ce624"} Oct 30 18:25:19.538188 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.538111 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.53809744 podStartE2EDuration="1.53809744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:19.536116916 +0000 UTC m=+1682.242302276" watchObservedRunningTime="2026-10-30 18:25:19.53809744 +0000 UTC m=+1682.244282797" Oct 30 18:25:19.874078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:19.874046 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:20.989004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:20.988959 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:25:26.530648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:26.530615 6966 generic.go:356] "Generic (PLEG): container finished" podID="eccd2c86-b51e-482f-9116-ddecf6a1b930" containerID="e30dc48b1e71f304cf89ed947b7f7ede4a7073dfbc9f0af43c57baefac002222" exitCode=137 Oct 30 18:25:26.531100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:26.530684 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" event={"ID":"eccd2c86-b51e-482f-9116-ddecf6a1b930","Type":"ContainerDied","Data":"e30dc48b1e71f304cf89ed947b7f7ede4a7073dfbc9f0af43c57baefac002222"} Oct 30 18:25:26.531100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:26.530992 6966 scope.go:122] "RemoveContainer" containerID="e30dc48b1e71f304cf89ed947b7f7ede4a7073dfbc9f0af43c57baefac002222" Oct 30 18:25:26.531348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:26.531333 6966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:25:27.539034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:27.538999 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" event={"ID":"eccd2c86-b51e-482f-9116-ddecf6a1b930","Type":"ContainerStarted","Data":"9ae74512572cdab9fcaa3b06b648cd8973b0f6030442996f18763bb5838138e0"} Oct 30 18:25:27.539578 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:27.539219 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:25:38.547560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:38.547534 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:25:47.199148 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:25:47.199116 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:25:55.796657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:25:55.796617 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:17.199248 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:26:17.199212 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:26:20.021694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:26:20.021647 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:21.068711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:26:21.068672 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:42.864401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:26:42.864366 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:26:47.199824 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:26:47.199787 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:26:59.501442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:26:59.501404 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:17.199618 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:27:17.199582 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:27:20.238473 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:20.238436 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:21.197408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:21.197370 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:25.770707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:25.770668 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.293755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:27.293719 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:27.675008 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:27.674978 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:27.675238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:27.675192 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="fefe5f17-9d89-4a89-953d-dd4c6fba9587" containerName="otc-container" containerID="cri-o://389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235" gracePeriod=30 Oct 30 18:27:27.940647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:27.940554 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:28.167512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:28.167455 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:29.182028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:29.181940 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:30.045816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.045791 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.175841 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.175816 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc\" (UniqueName: \"kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc\") pod \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " Oct 30 18:27:30.176004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.175879 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs\") pod \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " Oct 30 18:27:30.176004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.175911 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert\") pod \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " Oct 30 18:27:30.176004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.175954 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs\") pod \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " Oct 30 18:27:30.176004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.175981 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal\" (UniqueName: \"kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal\") pod \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " Oct 30 18:27:30.176168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.176012 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs\") pod \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\" (UID: \"fefe5f17-9d89-4a89-953d-dd4c6fba9587\") " Oct 30 18:27:30.176431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.176399 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal" pod "fefe5f17-9d89-4a89-953d-dd4c6fba9587" (UID: "fefe5f17-9d89-4a89-953d-dd4c6fba9587"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:30.178300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.178264 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert" pod "fefe5f17-9d89-4a89-953d-dd4c6fba9587" (UID: "fefe5f17-9d89-4a89-953d-dd4c6fba9587"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:30.178522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.178489 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs" pod "fefe5f17-9d89-4a89-953d-dd4c6fba9587" (UID: "fefe5f17-9d89-4a89-953d-dd4c6fba9587"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:30.178589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.178557 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs" pod "fefe5f17-9d89-4a89-953d-dd4c6fba9587" (UID: "fefe5f17-9d89-4a89-953d-dd4c6fba9587"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:30.178589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.178572 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc" pod "fefe5f17-9d89-4a89-953d-dd4c6fba9587" (UID: "fefe5f17-9d89-4a89-953d-dd4c6fba9587"). InnerVolumeSpecName "kube-api-access-7fjkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:30.178794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.178766 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs" pod "fefe5f17-9d89-4a89-953d-dd4c6fba9587" (UID: "fefe5f17-9d89-4a89-953d-dd4c6fba9587"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:30.180748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.180717 6966 generic.go:356] "Generic (PLEG): container finished" podID="fefe5f17-9d89-4a89-953d-dd4c6fba9587" containerID="389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235" exitCode=0 Oct 30 18:27:30.180840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.180761 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:27:30.180840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.180790 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fefe5f17-9d89-4a89-953d-dd4c6fba9587","Type":"ContainerDied","Data":"389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235"} Oct 30 18:27:30.180922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.180845 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"fefe5f17-9d89-4a89-953d-dd4c6fba9587","Type":"ContainerDied","Data":"7dbd3d10ae82b0bb98716cee8e0b76171711afc87748634b298d0582dc9ce624"} Oct 30 18:27:30.180922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.180869 6966 scope.go:122] "RemoveContainer" containerID="389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235" Oct 30 18:27:30.199911 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.199895 6966 scope.go:122] "RemoveContainer" containerID="389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235" Oct 30 18:27:30.200180 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:27:30.200169 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235\": container with ID starting with 389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235 not found: ID does not exist" containerID="389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235" Oct 30 18:27:30.200221 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.200189 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235"} err="failed to get container status \"389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235\": rpc error: code = NotFound desc = could not find container \"389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235\": container with ID starting with 389ac846fb1bee2be08777ed4f6ffcc404e77e80b054f4dd7dd966241b0b5235 not found: ID does not exist" Oct 30 18:27:30.237595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.237552 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:30.243381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.243350 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:27:30.277160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.277137 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fjkc\" (UniqueName: \"kubernetes.io/projected/fefe5f17-9d89-4a89-953d-dd4c6fba9587-kube-api-access-7fjkc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.277160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.277158 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.277287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.277170 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.277287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.277179 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.277287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.277187 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/fefe5f17-9d89-4a89-953d-dd4c6fba9587-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.277287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:30.277196 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fefe5f17-9d89-4a89-953d-dd4c6fba9587-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.411569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:31.411537 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefe5f17-9d89-4a89-953d-dd4c6fba9587" path="/var/lib/kubelet/pods/fefe5f17-9d89-4a89-953d-dd4c6fba9587/volumes" Oct 30 18:27:34.222699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:34.222665 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:39.326293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:39.326257 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:40.005406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:40.005373 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:47.199173 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:27:47.199136 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:27:50.543500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:50.543445 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:50.775748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:50.775718 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.517367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:52.517311 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:52.736494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:52.736420 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:53.867421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:53.867384 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:54.160869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:54.160771 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:27:54.405769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:27:54.405736 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:13.409664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:13.409624 6966 generic.go:356] "Generic (PLEG): container finished" podID="eccd2c86-b51e-482f-9116-ddecf6a1b930" containerID="9ae74512572cdab9fcaa3b06b648cd8973b0f6030442996f18763bb5838138e0" exitCode=137 Oct 30 18:28:13.410897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:13.410874 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" event={"ID":"eccd2c86-b51e-482f-9116-ddecf6a1b930","Type":"ContainerDied","Data":"9ae74512572cdab9fcaa3b06b648cd8973b0f6030442996f18763bb5838138e0"} Oct 30 18:28:13.410986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:13.410933 6966 scope.go:122] "RemoveContainer" containerID="e30dc48b1e71f304cf89ed947b7f7ede4a7073dfbc9f0af43c57baefac002222" Oct 30 18:28:13.411244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:13.411220 6966 scope.go:122] "RemoveContainer" containerID="9ae74512572cdab9fcaa3b06b648cd8973b0f6030442996f18763bb5838138e0" Oct 30 18:28:13.411495 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:28:13.411454 6966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-observability-5999d9c564-mm8f7_odh-observability-system(eccd2c86-b51e-482f-9116-ddecf6a1b930)\"" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" podUID="eccd2c86-b51e-482f-9116-ddecf6a1b930" Oct 30 18:28:17.199738 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:28:17.199701 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:28:18.545286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:18.545248 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:28:18.545912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:18.545749 6966 scope.go:122] "RemoveContainer" containerID="9ae74512572cdab9fcaa3b06b648cd8973b0f6030442996f18763bb5838138e0" Oct 30 18:28:18.546063 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:28:18.546039 6966 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"manager\" with CrashLoopBackOff: \"back-off 10s restarting failed container=manager pod=odh-observability-5999d9c564-mm8f7_odh-observability-system(eccd2c86-b51e-482f-9116-ddecf6a1b930)\"" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" podUID="eccd2c86-b51e-482f-9116-ddecf6a1b930" Oct 30 18:28:20.265922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:20.265890 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:21.320671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:21.320626 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:23.406000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:23.405973 6966 scope.go:122] "RemoveContainer" containerID="9ae74512572cdab9fcaa3b06b648cd8973b0f6030442996f18763bb5838138e0" Oct 30 18:28:24.469795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:24.469760 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" event={"ID":"eccd2c86-b51e-482f-9116-ddecf6a1b930","Type":"ContainerStarted","Data":"21403e438c3947357f84204a78e564eea2c751085f4abc718f99568d56c274de"} Oct 30 18:28:24.470200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:24.469959 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:28:35.439784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:35.439741 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:28:35.480303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:28:35.480283 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-observability-system/odh-observability-5999d9c564-mm8f7" Oct 30 18:28:47.198526 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:28:47.198491 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:29:17.199585 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:29:17.199501 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:29:20.383291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:29:20.383253 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:21.449435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:29:21.449396 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:33.347897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:29:33.347856 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:47.199323 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:29:47.199286 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:29:56.524927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:29:56.524887 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:29:56.746782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:29:56.746751 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:17.202296 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:30:17.199739 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:30:20.564052 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:30:20.564012 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:21.557396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:30:21.557363 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:30:47.200197 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:30:47.200117 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:31:17.199220 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:31:17.199169 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:31:20.628108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:31:20.628072 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:21.661727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:31:21.661693 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:34.938156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:31:34.938118 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:31:47.199735 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:31:47.199695 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:32:17.199657 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:32:17.199575 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:32:20.768773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:20.768740 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:21.782298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:21.782264 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:24.744510 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:24.744472 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:30.617581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:30.617533 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:32.538679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:32.538621 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:36.766765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:36.766724 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:36.975548 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:36.975509 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:37.176976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:37.176947 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:42.832494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:42.832441 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:43.063544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:43.063511 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:47.199204 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:32:47.199164 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:32:54.106983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:54.106944 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:54.307345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:54.307303 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:54.508803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:54.508699 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:56.100492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:56.100431 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:32:56.307267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:32:56.307216 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:02.197061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.197022 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:33:02.197562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.197545 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fefe5f17-9d89-4a89-953d-dd4c6fba9587" containerName="otc-container" Oct 30 18:33:02.197630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.197562 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefe5f17-9d89-4a89-953d-dd4c6fba9587" containerName="otc-container" Oct 30 18:33:02.197705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.197690 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefe5f17-9d89-4a89-953d-dd4c6fba9587" containerName="otc-container" Oct 30 18:33:02.201144 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.201123 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.226409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.226383 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:33:02.226409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.226383 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:33:02.226619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.226446 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:33:02.226619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.226559 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsdwv\"" Oct 30 18:33:02.226619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.226582 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:33:02.226825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.226628 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:33:02.227954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.227930 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.228083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.228043 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.228148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.228102 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.228241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.228218 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.228288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.228267 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.228365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.228343 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9xcl\" (UniqueName: \"kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.240366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.240340 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:33:02.329303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.329276 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.329434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.329310 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.329434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.329331 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9xcl\" (UniqueName: \"kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.329434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.329373 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.329603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.329440 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.329603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.329509 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.330217 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.330191 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.331863 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.331835 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.331985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.331933 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.331985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.331962 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.332105 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.332094 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.351867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.351846 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9xcl\" (UniqueName: \"kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl\") pod \"data-science-collector-collector-0\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.516454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.516385 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:33:02.827698 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.827659 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:33:02.832365 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:33:02.832331 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b370a0e_fb14_4a44_a4a2_196b3f66d01c.slice/crio-3f80864e27153783810040d3d238f11acae950d87954d1bfecc8c6a5ebd90c1c WatchSource:0}: Error finding container 3f80864e27153783810040d3d238f11acae950d87954d1bfecc8c6a5ebd90c1c: Status 404 returned error can't find the container with id 3f80864e27153783810040d3d238f11acae950d87954d1bfecc8c6a5ebd90c1c Oct 30 18:33:02.834485 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.834434 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:02.834572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.834516 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:33:02.912589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.912548 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5b370a0e-fb14-4a44-a4a2-196b3f66d01c","Type":"ContainerStarted","Data":"a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5"} Oct 30 18:33:02.912704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.912619 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5b370a0e-fb14-4a44-a4a2-196b3f66d01c","Type":"ContainerStarted","Data":"3f80864e27153783810040d3d238f11acae950d87954d1bfecc8c6a5ebd90c1c"} Oct 30 18:33:02.955038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:02.954979 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=0.954962167 podStartE2EDuration="954.962167ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:02.953849777 +0000 UTC m=+2145.660035133" watchObservedRunningTime="2026-10-30 18:33:02.954962167 +0000 UTC m=+2145.661147525" Oct 30 18:33:17.199760 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:33:17.199722 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:33:20.875560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:20.875524 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:21.915735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:21.915701 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:43.291061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:43.291021 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:43.501223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:33:43.501186 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:33:47.199374 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:33:47.199340 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:34:17.199606 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:34:17.199570 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:34:21.099020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:34:21.098982 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:22.030969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:34:22.030933 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:34:47.199769 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:34:47.199729 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:35:17.199682 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:35:17.199597 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:35:21.120620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:35:21.120578 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:22.154690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:35:22.154650 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:35:47.199850 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:35:47.199816 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:36:17.200362 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:36:17.200312 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:36:21.253522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:36:21.253486 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:22.288532 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:36:22.288495 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:47.199315 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:36:47.199280 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:36:52.362281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:36:52.362244 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:59.581846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:36:59.581813 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:36:59.817654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:36:59.817614 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:01.850479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:01.850422 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:17.199578 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:17.199542 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:37:20.314012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:20.313951 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:20.550718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:20.550686 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:21.378299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:21.378268 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:22.406388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:22.406359 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:26.372117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:26.372071 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:26.611683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:26.611646 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:27.841305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:27.841254 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:31.562342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:31.562295 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:31.783885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:31.783851 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:34.025231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:34.025201 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:44.206176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:44.206137 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:47.199852 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:47.199817 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:37:50.174361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:50.174312 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:50.391006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:50.390975 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:51.739956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.739920 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l"] Oct 30 18:37:51.740410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.740170 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="kube-rbac-proxy" containerID="cri-o://c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" gracePeriod=30 Oct 30 18:37:51.740410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.740245 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="prom-label-proxy" containerID="cri-o://d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" gracePeriod=30 Oct 30 18:37:51.740410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.740288 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="method-gate" containerID="cri-o://9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" gracePeriod=30 Oct 30 18:37:51.755397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.755369 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l"] Oct 30 18:37:51.755669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.755644 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="kube-rbac-proxy" containerID="cri-o://d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" gracePeriod=30 Oct 30 18:37:51.755755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.755708 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="method-gate" containerID="cri-o://b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" gracePeriod=30 Oct 30 18:37:51.755755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.755650 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="prom-label-proxy" containerID="cri-o://ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" gracePeriod=30 Oct 30 18:37:51.777562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.777536 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4"] Oct 30 18:37:51.777768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:51.777746 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" podUID="6b58ce80-1323-4e58-9235-4a32144cf1bf" containerName="korrel8r" containerID="cri-o://56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a" gracePeriod=30 Oct 30 18:37:52.010151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.010126 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:37:52.076781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.076752 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:37:52.087125 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.087093 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:37:52.137201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137124 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config\" (UniqueName: \"kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config\") pod \"f7eb340d-c73d-4ede-90eb-c17b4f595130\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137276 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config\") pod \"d012103e-2215-41a0-8217-61bb33ffa01a\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137298 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p\" (UniqueName: \"kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p\") pod \"f7eb340d-c73d-4ede-90eb-c17b4f595130\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137346 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2\" (UniqueName: \"kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2\") pod \"6b58ce80-1323-4e58-9235-4a32144cf1bf\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137364 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6\" (UniqueName: \"kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6\") pod \"d012103e-2215-41a0-8217-61bb33ffa01a\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137407 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls\" (UniqueName: \"kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls\") pod \"d012103e-2215-41a0-8217-61bb33ffa01a\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137433 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config\" (UniqueName: \"kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config\") pod \"6b58ce80-1323-4e58-9235-4a32144cf1bf\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " Oct 30 18:37:52.137505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137451 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca\") pod \"d012103e-2215-41a0-8217-61bb33ffa01a\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " Oct 30 18:37:52.137904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137509 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls\" (UniqueName: \"kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls\") pod \"f7eb340d-c73d-4ede-90eb-c17b4f595130\" (UID: \"f7eb340d-c73d-4ede-90eb-c17b4f595130\") " Oct 30 18:37:52.137904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137538 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls\" (UniqueName: \"kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls\") pod \"6b58ce80-1323-4e58-9235-4a32144cf1bf\" (UID: \"6b58ce80-1323-4e58-9235-4a32144cf1bf\") " Oct 30 18:37:52.137904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.137576 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca\") pod \"d012103e-2215-41a0-8217-61bb33ffa01a\" (UID: \"d012103e-2215-41a0-8217-61bb33ffa01a\") " Oct 30 18:37:52.138158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.138128 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca" pod "d012103e-2215-41a0-8217-61bb33ffa01a" (UID: "d012103e-2215-41a0-8217-61bb33ffa01a"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.138559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.138513 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config" pod "6b58ce80-1323-4e58-9235-4a32144cf1bf" (UID: "6b58ce80-1323-4e58-9235-4a32144cf1bf"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.138984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.138952 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca" pod "d012103e-2215-41a0-8217-61bb33ffa01a" (UID: "d012103e-2215-41a0-8217-61bb33ffa01a"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.139865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.139816 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config" pod "f7eb340d-c73d-4ede-90eb-c17b4f595130" (UID: "f7eb340d-c73d-4ede-90eb-c17b4f595130"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.139865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.139849 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config" pod "d012103e-2215-41a0-8217-61bb33ffa01a" (UID: "d012103e-2215-41a0-8217-61bb33ffa01a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.141686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.141660 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6" pod "d012103e-2215-41a0-8217-61bb33ffa01a" (UID: "d012103e-2215-41a0-8217-61bb33ffa01a"). InnerVolumeSpecName "kube-api-access-wr2n6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.143757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.143732 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2" pod "6b58ce80-1323-4e58-9235-4a32144cf1bf" (UID: "6b58ce80-1323-4e58-9235-4a32144cf1bf"). InnerVolumeSpecName "kube-api-access-nbnv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.148410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.148229 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p" pod "f7eb340d-c73d-4ede-90eb-c17b4f595130" (UID: "f7eb340d-c73d-4ede-90eb-c17b4f595130"). InnerVolumeSpecName "kube-api-access-qgs8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.148410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.148229 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls" pod "d012103e-2215-41a0-8217-61bb33ffa01a" (UID: "d012103e-2215-41a0-8217-61bb33ffa01a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.148767 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.148731 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls" pod "f7eb340d-c73d-4ede-90eb-c17b4f595130" (UID: "f7eb340d-c73d-4ede-90eb-c17b4f595130"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.152658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.152634 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls" pod "6b58ce80-1323-4e58-9235-4a32144cf1bf" (UID: "6b58ce80-1323-4e58-9235-4a32144cf1bf"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.204279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.204248 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:37:52.204790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.204758 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="alertmanager" containerID="cri-o://d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e" gracePeriod=120 Oct 30 18:37:52.204996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.204761 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="config-reloader" containerID="cri-o://ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de" gracePeriod=120 Oct 30 18:37:52.215249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.215211 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:37:52.215542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.215513 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="prometheus" containerID="cri-o://81422238d284177f6ad5e030885061bb0732f913662ed919bc2b9f565cfd8932" gracePeriod=600 Oct 30 18:37:52.215671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.215554 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="config-reloader" containerID="cri-o://50a2e119d7a5ad8c3ffcdf93d4c603c9c7e159dd614538f2f3daae2c3c0ffeee" gracePeriod=600 Oct 30 18:37:52.215830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.215800 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="thanos-sidecar" containerID="cri-o://ec9d260e94e1c8e9da0b336c84f697ebda42942ce2b5cccb2176663f371d2984" gracePeriod=600 Oct 30 18:37:52.219277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.219252 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:52.238959 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.238938 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6b58ce80-1323-4e58-9235-4a32144cf1bf-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.238961 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-prometheus-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.238972 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/f7eb340d-c73d-4ede-90eb-c17b4f595130-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.238979 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6b58ce80-1323-4e58-9235-4a32144cf1bf-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.238987 6966 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-client-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.238996 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/f7eb340d-c73d-4ede-90eb-c17b4f595130-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.239003 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d012103e-2215-41a0-8217-61bb33ffa01a-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.239011 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgs8p\" (UniqueName: \"kubernetes.io/projected/f7eb340d-c73d-4ede-90eb-c17b4f595130-kube-api-access-qgs8p\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.239019 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnv2\" (UniqueName: \"kubernetes.io/projected/6b58ce80-1323-4e58-9235-4a32144cf1bf-kube-api-access-nbnv2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.239027 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2n6\" (UniqueName: \"kubernetes.io/projected/d012103e-2215-41a0-8217-61bb33ffa01a-kube-api-access-wr2n6\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.239046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.239034 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/d012103e-2215-41a0-8217-61bb33ffa01a-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.388363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388330 6966 generic.go:356] "Generic (PLEG): container finished" podID="d012103e-2215-41a0-8217-61bb33ffa01a" containerID="d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" exitCode=0 Oct 30 18:37:52.388363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388359 6966 generic.go:356] "Generic (PLEG): container finished" podID="d012103e-2215-41a0-8217-61bb33ffa01a" containerID="9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" exitCode=0 Oct 30 18:37:52.388363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388366 6966 generic.go:356] "Generic (PLEG): container finished" podID="d012103e-2215-41a0-8217-61bb33ffa01a" containerID="c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" exitCode=0 Oct 30 18:37:52.388684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388406 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" Oct 30 18:37:52.388684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388412 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerDied","Data":"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff"} Oct 30 18:37:52.388684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388519 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerDied","Data":"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf"} Oct 30 18:37:52.388684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388561 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerDied","Data":"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f"} Oct 30 18:37:52.388684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388587 6966 scope.go:122] "RemoveContainer" containerID="d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" Oct 30 18:37:52.388684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.388599 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l" event={"ID":"d012103e-2215-41a0-8217-61bb33ffa01a","Type":"ContainerDied","Data":"d47a82e46f7a318233221b54fd2298bbf57eccf16a1b250a1edda762795f525a"} Oct 30 18:37:52.392974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.392954 6966 generic.go:356] "Generic (PLEG): container finished" podID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerID="ec9d260e94e1c8e9da0b336c84f697ebda42942ce2b5cccb2176663f371d2984" exitCode=0 Oct 30 18:37:52.392974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.392972 6966 generic.go:356] "Generic (PLEG): container finished" podID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerID="50a2e119d7a5ad8c3ffcdf93d4c603c9c7e159dd614538f2f3daae2c3c0ffeee" exitCode=0 Oct 30 18:37:52.393113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.392982 6966 generic.go:356] "Generic (PLEG): container finished" podID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerID="81422238d284177f6ad5e030885061bb0732f913662ed919bc2b9f565cfd8932" exitCode=0 Oct 30 18:37:52.393113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.393006 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerDied","Data":"ec9d260e94e1c8e9da0b336c84f697ebda42942ce2b5cccb2176663f371d2984"} Oct 30 18:37:52.393113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.393080 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerDied","Data":"50a2e119d7a5ad8c3ffcdf93d4c603c9c7e159dd614538f2f3daae2c3c0ffeee"} Oct 30 18:37:52.393271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.393120 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerDied","Data":"81422238d284177f6ad5e030885061bb0732f913662ed919bc2b9f565cfd8932"} Oct 30 18:37:52.394971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.394947 6966 generic.go:356] "Generic (PLEG): container finished" podID="6b58ce80-1323-4e58-9235-4a32144cf1bf" containerID="56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a" exitCode=0 Oct 30 18:37:52.395124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.394984 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" Oct 30 18:37:52.395124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.395060 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" event={"ID":"6b58ce80-1323-4e58-9235-4a32144cf1bf","Type":"ContainerDied","Data":"56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a"} Oct 30 18:37:52.395124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.395098 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4" event={"ID":"6b58ce80-1323-4e58-9235-4a32144cf1bf","Type":"ContainerDied","Data":"435659840d6cafda37e6aee47a451af5b991ec39b0b3375e80311dc9fb630881"} Oct 30 18:37:52.397209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397185 6966 generic.go:356] "Generic (PLEG): container finished" podID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerID="ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" exitCode=0 Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397212 6966 generic.go:356] "Generic (PLEG): container finished" podID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerID="b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" exitCode=0 Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397221 6966 generic.go:356] "Generic (PLEG): container finished" podID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerID="d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" exitCode=0 Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397263 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397295 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerDied","Data":"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde"} Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397350 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerDied","Data":"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4"} Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397392 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerDied","Data":"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec"} Oct 30 18:37:52.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.397427 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l" event={"ID":"f7eb340d-c73d-4ede-90eb-c17b4f595130","Type":"ContainerDied","Data":"3a49524b10b453bc171208fb1217c3ac75e3e41f769b6702690b46822a19508f"} Oct 30 18:37:52.400445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.400421 6966 generic.go:356] "Generic (PLEG): container finished" podID="ba07d3e9-2718-4195-977d-131737891144" containerID="ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de" exitCode=0 Oct 30 18:37:52.400573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.400505 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerDied","Data":"ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de"} Oct 30 18:37:52.404667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.404637 6966 scope.go:122] "RemoveContainer" containerID="9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" Oct 30 18:37:52.416611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.416588 6966 scope.go:122] "RemoveContainer" containerID="c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" Oct 30 18:37:52.425373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.425358 6966 scope.go:122] "RemoveContainer" containerID="d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" Oct 30 18:37:52.425684 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.425668 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": container with ID starting with d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff not found: ID does not exist" containerID="d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" Oct 30 18:37:52.425748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.425692 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff"} err="failed to get container status \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": rpc error: code = NotFound desc = could not find container \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": container with ID starting with d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff not found: ID does not exist" Oct 30 18:37:52.425748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.425709 6966 scope.go:122] "RemoveContainer" containerID="9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" Oct 30 18:37:52.425984 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.425960 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": container with ID starting with 9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf not found: ID does not exist" containerID="9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" Oct 30 18:37:52.426032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.425996 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf"} err="failed to get container status \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": rpc error: code = NotFound desc = could not find container \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": container with ID starting with 9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf not found: ID does not exist" Oct 30 18:37:52.426032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.426017 6966 scope.go:122] "RemoveContainer" containerID="c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" Oct 30 18:37:52.426296 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.426274 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": container with ID starting with c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f not found: ID does not exist" containerID="c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" Oct 30 18:37:52.426356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.426302 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f"} err="failed to get container status \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": rpc error: code = NotFound desc = could not find container \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": container with ID starting with c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f not found: ID does not exist" Oct 30 18:37:52.426356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.426320 6966 scope.go:122] "RemoveContainer" containerID="d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" Oct 30 18:37:52.426831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.426802 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff"} err="failed to get container status \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": rpc error: code = NotFound desc = could not find container \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": container with ID starting with d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff not found: ID does not exist" Oct 30 18:37:52.426900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.426832 6966 scope.go:122] "RemoveContainer" containerID="9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" Oct 30 18:37:52.427123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427098 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf"} err="failed to get container status \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": rpc error: code = NotFound desc = could not find container \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": container with ID starting with 9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf not found: ID does not exist" Oct 30 18:37:52.427181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427125 6966 scope.go:122] "RemoveContainer" containerID="c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" Oct 30 18:37:52.427376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427353 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f"} err="failed to get container status \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": rpc error: code = NotFound desc = could not find container \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": container with ID starting with c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f not found: ID does not exist" Oct 30 18:37:52.427437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427378 6966 scope.go:122] "RemoveContainer" containerID="d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff" Oct 30 18:37:52.427626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427607 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff"} err="failed to get container status \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": rpc error: code = NotFound desc = could not find container \"d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff\": container with ID starting with d3916902da5a6bce25429e7ace9ea9e7a5aa7756a0834a770134a1d39ec062ff not found: ID does not exist" Oct 30 18:37:52.427726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427628 6966 scope.go:122] "RemoveContainer" containerID="9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf" Oct 30 18:37:52.427885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427865 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf"} err="failed to get container status \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": rpc error: code = NotFound desc = could not find container \"9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf\": container with ID starting with 9c985f1f3d6eea1844505564aec46d8c6f51718e561c18ebfef801211890b6cf not found: ID does not exist" Oct 30 18:37:52.427938 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.427886 6966 scope.go:122] "RemoveContainer" containerID="c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f" Oct 30 18:37:52.428169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.428146 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f"} err="failed to get container status \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": rpc error: code = NotFound desc = could not find container \"c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f\": container with ID starting with c15b33b89fa0d1ce4e4a618c58da7508f2af28a49cb79baf15032deca7f2f27f not found: ID does not exist" Oct 30 18:37:52.428239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.428170 6966 scope.go:122] "RemoveContainer" containerID="56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a" Oct 30 18:37:52.435832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.435815 6966 scope.go:122] "RemoveContainer" containerID="56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a" Oct 30 18:37:52.436092 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.436071 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a\": container with ID starting with 56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a not found: ID does not exist" containerID="56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a" Oct 30 18:37:52.436152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.436100 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a"} err="failed to get container status \"56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a\": rpc error: code = NotFound desc = could not find container \"56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a\": container with ID starting with 56211683144b8bb0b36635b800f312e83da914aa0766522ad834c5c83b0eed0a not found: ID does not exist" Oct 30 18:37:52.436152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.436118 6966 scope.go:122] "RemoveContainer" containerID="ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" Oct 30 18:37:52.443811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.443759 6966 scope.go:122] "RemoveContainer" containerID="b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" Oct 30 18:37:52.451027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.451008 6966 scope.go:122] "RemoveContainer" containerID="d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" Oct 30 18:37:52.457322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.457291 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:52.459193 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.459163 6966 scope.go:122] "RemoveContainer" containerID="ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" Oct 30 18:37:52.459497 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.459439 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": container with ID starting with ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde not found: ID does not exist" containerID="ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" Oct 30 18:37:52.459585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.459506 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde"} err="failed to get container status \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": rpc error: code = NotFound desc = could not find container \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": container with ID starting with ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde not found: ID does not exist" Oct 30 18:37:52.459585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.459527 6966 scope.go:122] "RemoveContainer" containerID="b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" Oct 30 18:37:52.459825 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.459803 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": container with ID starting with b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4 not found: ID does not exist" containerID="b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" Oct 30 18:37:52.459916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.459828 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4"} err="failed to get container status \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": rpc error: code = NotFound desc = could not find container \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": container with ID starting with b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4 not found: ID does not exist" Oct 30 18:37:52.459916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.459847 6966 scope.go:122] "RemoveContainer" containerID="d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" Oct 30 18:37:52.460132 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:52.460115 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": container with ID starting with d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec not found: ID does not exist" containerID="d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" Oct 30 18:37:52.460186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460135 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec"} err="failed to get container status \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": rpc error: code = NotFound desc = could not find container \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": container with ID starting with d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec not found: ID does not exist" Oct 30 18:37:52.460186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460148 6966 scope.go:122] "RemoveContainer" containerID="ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" Oct 30 18:37:52.460412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460392 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde"} err="failed to get container status \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": rpc error: code = NotFound desc = could not find container \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": container with ID starting with ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde not found: ID does not exist" Oct 30 18:37:52.460490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460420 6966 scope.go:122] "RemoveContainer" containerID="b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" Oct 30 18:37:52.460694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460678 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4"} err="failed to get container status \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": rpc error: code = NotFound desc = could not find container \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": container with ID starting with b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4 not found: ID does not exist" Oct 30 18:37:52.460694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460694 6966 scope.go:122] "RemoveContainer" containerID="d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" Oct 30 18:37:52.460905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460886 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec"} err="failed to get container status \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": rpc error: code = NotFound desc = could not find container \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": container with ID starting with d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec not found: ID does not exist" Oct 30 18:37:52.460979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.460906 6966 scope.go:122] "RemoveContainer" containerID="ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde" Oct 30 18:37:52.461180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.461160 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde"} err="failed to get container status \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": rpc error: code = NotFound desc = could not find container \"ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde\": container with ID starting with ff638abce862b7c4888d3e8983a8b8ab4dcf50e5f6261b27940eccc1b8058dde not found: ID does not exist" Oct 30 18:37:52.461228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.461181 6966 scope.go:122] "RemoveContainer" containerID="b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4" Oct 30 18:37:52.461380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.461362 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4"} err="failed to get container status \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": rpc error: code = NotFound desc = could not find container \"b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4\": container with ID starting with b38718e44253e807f878ae10cf1868a12ecd2be78a9f105ae96da40be599e9e4 not found: ID does not exist" Oct 30 18:37:52.461449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.461381 6966 scope.go:122] "RemoveContainer" containerID="d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec" Oct 30 18:37:52.461612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.461588 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec"} err="failed to get container status \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": rpc error: code = NotFound desc = could not find container \"d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec\": container with ID starting with d079935ce51be9a3a14231de8f9fad6e51cc30bb600a2c4e3826abbf9077f9ec not found: ID does not exist" Oct 30 18:37:52.466758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.466740 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l"] Oct 30 18:37:52.470656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.470617 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-77x5l"] Oct 30 18:37:52.477271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.477254 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:37:52.502498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.502476 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l"] Oct 30 18:37:52.509336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.509318 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6c42l"] Oct 30 18:37:52.540787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.540761 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.540884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.540816 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.540884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.540857 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.540985 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541082 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541124 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541182 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541258 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541226 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541266 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541299 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541353 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541451 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.541518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541501 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\" (UID: \"d7047b0d-c1f3-4d8b-9009-92ff228a86f1\") " Oct 30 18:37:52.542246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.541880 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.542246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.542199 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.543279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.542824 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:52.544323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.544293 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.545439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.545419 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4"] Oct 30 18:37:52.545605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.545529 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.545690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.545600 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "kube-api-access-b5m7j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.545758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.545726 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:52.546028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.546009 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:52.546188 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.546162 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.547232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.547190 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.547740 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.547656 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.550895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.550874 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:37:52.554387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.554366 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zkgv4"] Oct 30 18:37:52.558256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.558235 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config" pod "d7047b0d-c1f3-4d8b-9009-92ff228a86f1" (UID: "d7047b0d-c1f3-4d8b-9009-92ff228a86f1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:52.642358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642283 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642316 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642334 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642347 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642361 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5m7j\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-kube-api-access-b5m7j\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642375 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642412 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642425 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642435 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642444 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642453 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642478 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.642655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.642488 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d7047b0d-c1f3-4d8b-9009-92ff228a86f1-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:52.660869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.660849 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:37:52.703330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.703300 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:52.743299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.743275 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-83df9dfad9fd2ac35dca1441af096f35871cdbdee4927aba3c5e3091d60e99b5\"" Oct 30 18:37:52.826270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.825982 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:37:52.826270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.826226 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="5b370a0e-fb14-4a44-a4a2-196b3f66d01c" containerName="otc-container" containerID="cri-o://a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5" gracePeriod=30 Oct 30 18:37:52.936283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:52.936247 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:53.104040 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.104017 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:37:53.106148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.106127 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:53.167029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.166985 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:53.248630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248557 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.248630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248608 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs\") pod \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " Oct 30 18:37:53.248630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248632 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248662 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248723 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248762 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248788 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248821 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs\") pod \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248854 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert\") pod \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " Oct 30 18:37:53.248908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248890 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets\") pod \"ba07d3e9-2718-4195-977d-131737891144\" (UID: \"ba07d3e9-2718-4195-977d-131737891144\") " Oct 30 18:37:53.249268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248932 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl\" (UniqueName: \"kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl\") pod \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " Oct 30 18:37:53.249268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248967 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs\") pod \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " Oct 30 18:37:53.249268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.248998 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal\") pod \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\" (UID: \"5b370a0e-fb14-4a44-a4a2-196b3f66d01c\") " Oct 30 18:37:53.249268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.249069 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:53.249493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.249374 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal" pod "5b370a0e-fb14-4a44-a4a2-196b3f66d01c" (UID: "5b370a0e-fb14-4a44-a4a2-196b3f66d01c"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:53.249808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.249631 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.249808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.249687 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.251505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.251445 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "kube-api-access-7cc4m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:53.251752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.251709 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs" pod "5b370a0e-fb14-4a44-a4a2-196b3f66d01c" (UID: "5b370a0e-fb14-4a44-a4a2-196b3f66d01c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.251938 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.251912 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:53.252534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.252508 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert" pod "5b370a0e-fb14-4a44-a4a2-196b3f66d01c" (UID: "5b370a0e-fb14-4a44-a4a2-196b3f66d01c"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.253231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.253209 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.253231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.253229 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs" pod "5b370a0e-fb14-4a44-a4a2-196b3f66d01c" (UID: "5b370a0e-fb14-4a44-a4a2-196b3f66d01c"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.254248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.254219 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:37:53.254248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.254216 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl" pod "5b370a0e-fb14-4a44-a4a2-196b3f66d01c" (UID: "5b370a0e-fb14-4a44-a4a2-196b3f66d01c"). InnerVolumeSpecName "kube-api-access-g9xcl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:53.254906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.254891 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs" pod "5b370a0e-fb14-4a44-a4a2-196b3f66d01c" (UID: "5b370a0e-fb14-4a44-a4a2-196b3f66d01c"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.257860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.257838 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.265121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.265100 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config" pod "ba07d3e9-2718-4195-977d-131737891144" (UID: "ba07d3e9-2718-4195-977d-131737891144"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:53.350043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350021 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ba07d3e9-2718-4195-977d-131737891144-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350042 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350053 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350062 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350072 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350082 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9xcl\" (UniqueName: \"kubernetes.io/projected/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-kube-api-access-g9xcl\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350091 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350099 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cc4m\" (UniqueName: \"kubernetes.io/projected/ba07d3e9-2718-4195-977d-131737891144-kube-api-access-7cc4m\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350107 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b370a0e-fb14-4a44-a4a2-196b3f66d01c-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350115 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.350171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.350124 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ba07d3e9-2718-4195-977d-131737891144-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:37:53.408049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.408024 6966 generic.go:356] "Generic (PLEG): container finished" podID="5b370a0e-fb14-4a44-a4a2-196b3f66d01c" containerID="a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5" exitCode=0 Oct 30 18:37:53.408176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.408155 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:53.410160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.410139 6966 generic.go:356] "Generic (PLEG): container finished" podID="ba07d3e9-2718-4195-977d-131737891144" containerID="d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e" exitCode=0 Oct 30 18:37:53.410286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.410269 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:37:53.412976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.412950 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b58ce80-1323-4e58-9235-4a32144cf1bf" path="/var/lib/kubelet/pods/6b58ce80-1323-4e58-9235-4a32144cf1bf/volumes" Oct 30 18:37:53.413541 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.413518 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" path="/var/lib/kubelet/pods/d012103e-2215-41a0-8217-61bb33ffa01a/volumes" Oct 30 18:37:53.413647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.413561 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:37:53.414211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.414190 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" path="/var/lib/kubelet/pods/f7eb340d-c73d-4ede-90eb-c17b4f595130/volumes" Oct 30 18:37:53.415453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.415432 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5b370a0e-fb14-4a44-a4a2-196b3f66d01c","Type":"ContainerDied","Data":"a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5"} Oct 30 18:37:53.415551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.415494 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"5b370a0e-fb14-4a44-a4a2-196b3f66d01c","Type":"ContainerDied","Data":"3f80864e27153783810040d3d238f11acae950d87954d1bfecc8c6a5ebd90c1c"} Oct 30 18:37:53.415551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.415520 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerDied","Data":"d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e"} Oct 30 18:37:53.415667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.415550 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ba07d3e9-2718-4195-977d-131737891144","Type":"ContainerDied","Data":"544be53c35b7f6a97bf265f4e9b35b0a1d59302ac194b73e7d2326c62c7124e8"} Oct 30 18:37:53.415667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.415585 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"d7047b0d-c1f3-4d8b-9009-92ff228a86f1","Type":"ContainerDied","Data":"057e191a8287c00733020144447fc2c3711d7d232b6b13aa13f6160fdeb2ef37"} Oct 30 18:37:53.415667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.415651 6966 scope.go:122] "RemoveContainer" containerID="a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5" Oct 30 18:37:53.430052 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.430038 6966 scope.go:122] "RemoveContainer" containerID="a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5" Oct 30 18:37:53.430357 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:53.430332 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5\": container with ID starting with a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5 not found: ID does not exist" containerID="a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5" Oct 30 18:37:53.430425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.430367 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5"} err="failed to get container status \"a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5\": rpc error: code = NotFound desc = could not find container \"a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5\": container with ID starting with a43549d4da52c0e6844c9fb27d6d26a61dd94449d5c29f7d5f377eac62f0dfb5 not found: ID does not exist" Oct 30 18:37:53.430425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.430389 6966 scope.go:122] "RemoveContainer" containerID="ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de" Oct 30 18:37:53.439604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.439585 6966 scope.go:122] "RemoveContainer" containerID="d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e" Oct 30 18:37:53.446999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.446972 6966 scope.go:122] "RemoveContainer" containerID="79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a" Oct 30 18:37:53.454411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.454392 6966 scope.go:122] "RemoveContainer" containerID="ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de" Oct 30 18:37:53.454676 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:53.454658 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de\": container with ID starting with ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de not found: ID does not exist" containerID="ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de" Oct 30 18:37:53.454741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.454682 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de"} err="failed to get container status \"ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de\": rpc error: code = NotFound desc = could not find container \"ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de\": container with ID starting with ffbce6e7fac9bb8774696e1df27696150ab83d444ec412bfac2a48eca4fdc5de not found: ID does not exist" Oct 30 18:37:53.454741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.454699 6966 scope.go:122] "RemoveContainer" containerID="d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e" Oct 30 18:37:53.454925 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:53.454910 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e\": container with ID starting with d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e not found: ID does not exist" containerID="d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e" Oct 30 18:37:53.454970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.454930 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e"} err="failed to get container status \"d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e\": rpc error: code = NotFound desc = could not find container \"d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e\": container with ID starting with d7966b6962e9a41a6aa97bf01571162aa9ca4fc55437314d80c37509481dfd8e not found: ID does not exist" Oct 30 18:37:53.454970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.454944 6966 scope.go:122] "RemoveContainer" containerID="79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a" Oct 30 18:37:53.455158 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:37:53.455138 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a\": container with ID starting with 79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a not found: ID does not exist" containerID="79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a" Oct 30 18:37:53.455216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.455167 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a"} err="failed to get container status \"79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a\": rpc error: code = NotFound desc = could not find container \"79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a\": container with ID starting with 79190237479c86bdc42f93cd6fabaabebc77dd44b379a5e152bade3f3b92156a not found: ID does not exist" Oct 30 18:37:53.455216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.455188 6966 scope.go:122] "RemoveContainer" containerID="ec9d260e94e1c8e9da0b336c84f697ebda42942ce2b5cccb2176663f371d2984" Oct 30 18:37:53.462356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.462339 6966 scope.go:122] "RemoveContainer" containerID="50a2e119d7a5ad8c3ffcdf93d4c603c9c7e159dd614538f2f3daae2c3c0ffeee" Oct 30 18:37:53.469931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.469916 6966 scope.go:122] "RemoveContainer" containerID="81422238d284177f6ad5e030885061bb0732f913662ed919bc2b9f565cfd8932" Oct 30 18:37:53.477684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.477667 6966 scope.go:122] "RemoveContainer" containerID="d932f41232be0464d4908e0fee90e0a82a8e3f931fba1c0f1a66e0cfbc2fa0d9" Oct 30 18:37:53.486615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.486595 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:37:53.492606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.492581 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:37:53.524634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.524585 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:37:53.529225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.529206 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:37:53.562796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.562764 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:37:53.568084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.568053 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:37:53.817524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:53.817490 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:54.033577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:54.033541 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:55.412927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.412897 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b370a0e-fb14-4a44-a4a2-196b3f66d01c" path="/var/lib/kubelet/pods/5b370a0e-fb14-4a44-a4a2-196b3f66d01c/volumes" Oct 30 18:37:55.413281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.413268 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba07d3e9-2718-4195-977d-131737891144" path="/var/lib/kubelet/pods/ba07d3e9-2718-4195-977d-131737891144/volumes" Oct 30 18:37:55.413798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.413783 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" path="/var/lib/kubelet/pods/d7047b0d-c1f3-4d8b-9009-92ff228a86f1/volumes" Oct 30 18:37:55.877903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.877871 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-27dft"] Oct 30 18:37:55.878199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878187 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="prom-label-proxy" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878200 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="prom-label-proxy" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878218 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="config-reloader" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878223 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="config-reloader" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878230 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="init-config-reloader" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878234 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="init-config-reloader" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878241 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="prometheus" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878246 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="prometheus" Oct 30 18:37:55.878253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878253 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="method-gate" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878257 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="method-gate" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878264 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="thanos-sidecar" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878268 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="thanos-sidecar" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878275 6966 cpu_manager.go:414] "containerMap: removing container" podUID="5b370a0e-fb14-4a44-a4a2-196b3f66d01c" containerName="otc-container" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878279 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b370a0e-fb14-4a44-a4a2-196b3f66d01c" containerName="otc-container" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878288 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="kube-rbac-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878292 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="kube-rbac-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878307 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="init-config-reloader" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878311 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="init-config-reloader" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878317 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="config-reloader" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878322 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="config-reloader" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878330 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="kube-rbac-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878334 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="kube-rbac-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878346 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="prom-label-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878350 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="prom-label-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878355 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="alertmanager" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878360 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="alertmanager" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878370 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="method-gate" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878374 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="method-gate" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878381 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6b58ce80-1323-4e58-9235-4a32144cf1bf" containerName="korrel8r" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878385 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b58ce80-1323-4e58-9235-4a32144cf1bf" containerName="korrel8r" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878430 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="config-reloader" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878437 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="kube-rbac-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878443 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="prom-label-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878449 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba07d3e9-2718-4195-977d-131737891144" containerName="alertmanager" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878454 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="method-gate" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878472 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b58ce80-1323-4e58-9235-4a32144cf1bf" containerName="korrel8r" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878478 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="prom-label-proxy" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878484 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="config-reloader" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878488 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="prometheus" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878495 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eb340d-c73d-4ede-90eb-c17b4f595130" containerName="method-gate" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878500 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7047b0d-c1f3-4d8b-9009-92ff228a86f1" containerName="thanos-sidecar" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878505 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b370a0e-fb14-4a44-a4a2-196b3f66d01c" containerName="otc-container" Oct 30 18:37:55.878519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.878510 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d012103e-2215-41a0-8217-61bb33ffa01a" containerName="kube-rbac-proxy" Oct 30 18:37:55.883757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.883740 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:55.897562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.897540 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:37:55.908907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.908880 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-7ktjd\"" Oct 30 18:37:55.909037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.908941 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:37:55.909037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.908944 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:37:55.909037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.909006 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:37:55.910315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.910295 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-27dft"] Oct 30 18:37:55.939904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.939882 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:55.971341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.971315 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:55.971445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.971342 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:55.971445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:55.971368 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkbmw\" (UniqueName: \"kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.072077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.072055 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.072203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.072087 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.072203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.072118 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkbmw\" (UniqueName: \"kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.072846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.072827 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.074762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.074744 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.094858 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.094840 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkbmw\" (UniqueName: \"kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw\") pod \"korrel8r-6644769df5-27dft\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.160130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.160068 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:56.196523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.196497 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:56.491213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.491146 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-27dft"] Oct 30 18:37:56.496477 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:37:56.496435 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776d6ee5_e229_4a6b_8fc8_850f0e85bb1e.slice/crio-e071377fc26c3e78475e4383d7a083dd774accbb2a8fb7b8b26309d9cfbbd4eb WatchSource:0}: Error finding container e071377fc26c3e78475e4383d7a083dd774accbb2a8fb7b8b26309d9cfbbd4eb: Status 404 returned error can't find the container with id e071377fc26c3e78475e4383d7a083dd774accbb2a8fb7b8b26309d9cfbbd4eb Oct 30 18:37:56.578788 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.578760 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:37:56.582480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.582440 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.601290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.601270 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:37:56.601863 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.601841 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:37:56.601950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.601861 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:37:56.606450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.606429 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:37:56.678516 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.678478 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.678697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.678616 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.678697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.678654 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.678782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.678765 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7t8\" (UniqueName: \"kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.723954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.723923 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:56.779345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.779283 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.779345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.779330 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.779580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.779364 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.779580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.779405 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7t8\" (UniqueName: \"kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.780074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.780053 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.781792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.781770 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.781878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.781809 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.801665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.801643 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7t8\" (UniqueName: \"kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8\") pod \"data-science-collector-collector-0\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.895811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.895784 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:37:56.942060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:56.942029 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:57.191512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.191482 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:37:57.196531 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:37:57.196504 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3006e4_be5e_4af6_a6fe_0e79b2630f82.slice/crio-39cdd556cb0c3736b2944a0ff3634dea1f10b841bbbc6e66e65045bd3fb41407 WatchSource:0}: Error finding container 39cdd556cb0c3736b2944a0ff3634dea1f10b841bbbc6e66e65045bd3fb41407: Status 404 returned error can't find the container with id 39cdd556cb0c3736b2944a0ff3634dea1f10b841bbbc6e66e65045bd3fb41407 Oct 30 18:37:57.198782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.198746 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:37:57.198867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.198805 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:37:57.442055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.441966 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a3006e4-be5e-4af6-a6fe-0e79b2630f82","Type":"ContainerStarted","Data":"59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e"} Oct 30 18:37:57.442055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.442023 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a3006e4-be5e-4af6-a6fe-0e79b2630f82","Type":"ContainerStarted","Data":"39cdd556cb0c3736b2944a0ff3634dea1f10b841bbbc6e66e65045bd3fb41407"} Oct 30 18:37:57.443439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.443411 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" event={"ID":"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e","Type":"ContainerStarted","Data":"df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340"} Oct 30 18:37:57.443577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.443451 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" event={"ID":"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e","Type":"ContainerStarted","Data":"e071377fc26c3e78475e4383d7a083dd774accbb2a8fb7b8b26309d9cfbbd4eb"} Oct 30 18:37:57.443577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.443518 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:37:57.490455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.490299 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=1.490286239 podStartE2EDuration="1.490286239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:57.489055399 +0000 UTC m=+2440.195240768" watchObservedRunningTime="2026-10-30 18:37:57.490286239 +0000 UTC m=+2440.196471596" Oct 30 18:37:57.532383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.527605 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" podStartSLOduration=2.527587975 podStartE2EDuration="2.527587975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:57.527081119 +0000 UTC m=+2440.233266476" watchObservedRunningTime="2026-10-30 18:37:57.527587975 +0000 UTC m=+2440.233773335" Oct 30 18:37:57.762040 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.761942 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:37:57.766161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.766141 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.780523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780500 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-mtls\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780500 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-serving-cabundle\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780529 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-ca-bundle\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780500 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780520 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway-serving-cert\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780531 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-gateway\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780594 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-dockercfg-xk9pn\"" Oct 30 18:37:57.780627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.780603 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempomonolithic-config\"" Oct 30 18:37:57.793254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.793232 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:37:57.888589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888561 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888607 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888636 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888726 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czkmw\" (UniqueName: \"kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888763 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888834 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888859 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.888959 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.888931 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.920853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.920822 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:37:57.989435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989411 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989441 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989483 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czkmw\" (UniqueName: \"kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989510 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989546 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989568 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989600 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.989844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.989651 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.990242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.990218 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.990392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.990269 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.990519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.990304 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.990701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.990538 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.992244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.992225 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.992336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.992248 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:57.992439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:57.992417 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:58.012126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:58.012081 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czkmw\" (UniqueName: \"kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:37:58.141006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:37:58.140973 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:06.859807 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:06.859771 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:06.886853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:06.886827 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") device path: \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:06.960803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:06.960777 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:06.960910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:06.960884 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") DevicePath \"\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:06.974518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:06.974498 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") DevicePath \"csi-f1d243cd1eff0987a64de94f5fb75fb003a2545b22342d982b5bcfa8f4a85c68\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:07.075087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:07.075060 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/a60e9ebe7e25cad823165cd28aa1946aee7872f3600d53c49936a4b9d631202d/globalmount\"" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:07.101039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:07.101011 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"tempo-data-science-tempomonolithic-0\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:07.383596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:07.383569 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:07.729102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:07.729015 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:38:07.735897 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:38:07.735870 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aae657c_c800_4102_ba42_9aba4f1113c3.slice/crio-3cafe95dbb6040991097d5f29d2eeb8d63a0a8ba4f52e4af7a9e837103bd7ca7 WatchSource:0}: Error finding container 3cafe95dbb6040991097d5f29d2eeb8d63a0a8ba4f52e4af7a9e837103bd7ca7: Status 404 returned error can't find the container with id 3cafe95dbb6040991097d5f29d2eeb8d63a0a8ba4f52e4af7a9e837103bd7ca7 Oct 30 18:38:08.450568 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.450536 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:38:08.498179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498152 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerStarted","Data":"0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d"} Oct 30 18:38:08.498343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498312 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:08.498401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498346 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerStarted","Data":"d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f"} Oct 30 18:38:08.498401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498366 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerStarted","Data":"6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4"} Oct 30 18:38:08.498401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498390 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:08.498543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498409 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:08.498543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.498423 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerStarted","Data":"3cafe95dbb6040991097d5f29d2eeb8d63a0a8ba4f52e4af7a9e837103bd7ca7"} Oct 30 18:38:08.500832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.500807 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:08.507417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.507397 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:08.545417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:08.545361 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podStartSLOduration=11.545344146 podStartE2EDuration="11.545344146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:37:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:08.542526343 +0000 UTC m=+2451.248711700" watchObservedRunningTime="2026-10-30 18:38:08.545344146 +0000 UTC m=+2451.251529505" Oct 30 18:38:17.198827 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:17.198774 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:38:21.517398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:21.517366 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:22.536189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:22.536152 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:30.510922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:30.510874 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:38:40.510639 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:40.510585 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:38:47.198820 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:47.198777 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:38:50.510394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:50.510358 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:51.956242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:51.956211 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-27dft"] Oct 30 18:38:51.956666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:51.956394 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" podUID="776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" containerName="korrel8r" containerID="cri-o://df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340" gracePeriod=30 Oct 30 18:38:52.202503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.202480 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:38:52.311100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.311081 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw\" (UniqueName: \"kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw\") pod \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " Oct 30 18:38:52.311257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.311121 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config\" (UniqueName: \"kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config\") pod \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " Oct 30 18:38:52.311257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.311152 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls\" (UniqueName: \"kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls\") pod \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\" (UID: \"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e\") " Oct 30 18:38:52.311564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.311523 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config" pod "776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" (UID: "776d6ee5-e229-4a6b-8fc8-850f0e85bb1e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:52.313140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.313110 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw" pod "776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" (UID: "776d6ee5-e229-4a6b-8fc8-850f0e85bb1e"). InnerVolumeSpecName "kube-api-access-vkbmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:52.313243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.313192 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls" pod "776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" (UID: "776d6ee5-e229-4a6b-8fc8-850f0e85bb1e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.347961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.347205 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:38:52.347961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.347540 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="8a3006e4-be5e-4af6-a6fe-0e79b2630f82" containerName="otc-container" containerID="cri-o://59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e" gracePeriod=30 Oct 30 18:38:52.412832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.412802 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkbmw\" (UniqueName: \"kubernetes.io/projected/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-kube-api-access-vkbmw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.412832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.412826 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.412832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.412836 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.542947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.542912 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:52.584953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.584930 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:52.714625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714598 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs\") pod \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " Oct 30 18:38:52.714625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714637 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs\") pod \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " Oct 30 18:38:52.714888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714659 6966 generic.go:356] "Generic (PLEG): container finished" podID="776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" containerID="df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340" exitCode=0 Oct 30 18:38:52.714888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714677 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8\" (UniqueName: \"kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8\") pod \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " Oct 30 18:38:52.714888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714732 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" Oct 30 18:38:52.714888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714770 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal\") pod \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\" (UID: \"8a3006e4-be5e-4af6-a6fe-0e79b2630f82\") " Oct 30 18:38:52.714888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714770 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" event={"ID":"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e","Type":"ContainerDied","Data":"df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340"} Oct 30 18:38:52.714888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714869 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-27dft" event={"ID":"776d6ee5-e229-4a6b-8fc8-850f0e85bb1e","Type":"ContainerDied","Data":"e071377fc26c3e78475e4383d7a083dd774accbb2a8fb7b8b26309d9cfbbd4eb"} Oct 30 18:38:52.715211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.714918 6966 scope.go:122] "RemoveContainer" containerID="df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340" Oct 30 18:38:52.715211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.715148 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal" pod "8a3006e4-be5e-4af6-a6fe-0e79b2630f82" (UID: "8a3006e4-be5e-4af6-a6fe-0e79b2630f82"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:52.717611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.717573 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs" pod "8a3006e4-be5e-4af6-a6fe-0e79b2630f82" (UID: "8a3006e4-be5e-4af6-a6fe-0e79b2630f82"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.717611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.717591 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8" pod "8a3006e4-be5e-4af6-a6fe-0e79b2630f82" (UID: "8a3006e4-be5e-4af6-a6fe-0e79b2630f82"). InnerVolumeSpecName "kube-api-access-mv7t8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:52.717611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.717598 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs" pod "8a3006e4-be5e-4af6-a6fe-0e79b2630f82" (UID: "8a3006e4-be5e-4af6-a6fe-0e79b2630f82"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:52.718668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.718642 6966 generic.go:356] "Generic (PLEG): container finished" podID="8a3006e4-be5e-4af6-a6fe-0e79b2630f82" containerID="59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e" exitCode=0 Oct 30 18:38:52.718773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.718679 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a3006e4-be5e-4af6-a6fe-0e79b2630f82","Type":"ContainerDied","Data":"59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e"} Oct 30 18:38:52.718773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.718695 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:38:52.718773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.718718 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"8a3006e4-be5e-4af6-a6fe-0e79b2630f82","Type":"ContainerDied","Data":"39cdd556cb0c3736b2944a0ff3634dea1f10b841bbbc6e66e65045bd3fb41407"} Oct 30 18:38:52.726180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.726162 6966 scope.go:122] "RemoveContainer" containerID="df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340" Oct 30 18:38:52.726524 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:52.726505 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340\": container with ID starting with df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340 not found: ID does not exist" containerID="df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340" Oct 30 18:38:52.726600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.726537 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340"} err="failed to get container status \"df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340\": rpc error: code = NotFound desc = could not find container \"df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340\": container with ID starting with df346345d651e6add61a27589f5aa0987f47ee41ca049b31789e5b029755d340 not found: ID does not exist" Oct 30 18:38:52.726600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.726561 6966 scope.go:122] "RemoveContainer" containerID="59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e" Oct 30 18:38:52.735179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.735158 6966 scope.go:122] "RemoveContainer" containerID="59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e" Oct 30 18:38:52.735615 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:52.735495 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e\": container with ID starting with 59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e not found: ID does not exist" containerID="59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e" Oct 30 18:38:52.735615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.735529 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e"} err="failed to get container status \"59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e\": rpc error: code = NotFound desc = could not find container \"59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e\": container with ID starting with 59157ff65e584145b19bcc6f5cdb14a7a45b4689cf153b5a98936c9c3e00ec4e not found: ID does not exist" Oct 30 18:38:52.768652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.768627 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-27dft"] Oct 30 18:38:52.775687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.775168 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-27dft"] Oct 30 18:38:52.793797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.793767 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:38:52.794049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.794022 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo" containerID="cri-o://6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4" gracePeriod=30 Oct 30 18:38:52.794220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.794200 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway-opa" containerID="cri-o://0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d" gracePeriod=30 Oct 30 18:38:52.794322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.794286 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway" containerID="cri-o://d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f" gracePeriod=30 Oct 30 18:38:52.798053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.798015 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:52.811681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.811661 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:38:52.815600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.815580 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.815680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.815602 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.815680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.815612 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.815680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.815621 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7t8\" (UniqueName: \"kubernetes.io/projected/8a3006e4-be5e-4af6-a6fe-0e79b2630f82-kube-api-access-mv7t8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:52.818738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:52.818719 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:38:53.028609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.028579 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:53.411471 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.411436 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" path="/var/lib/kubelet/pods/776d6ee5-e229-4a6b-8fc8-850f0e85bb1e/volumes" Oct 30 18:38:53.411851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.411837 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3006e4-be5e-4af6-a6fe-0e79b2630f82" path="/var/lib/kubelet/pods/8a3006e4-be5e-4af6-a6fe-0e79b2630f82/volumes" Oct 30 18:38:53.498676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.498640 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.89:8083/ready\": dial tcp 10.134.0.89:8083: connect: connection refused" Oct 30 18:38:53.502324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.502298 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:38:53.726999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.726923 6966 generic.go:356] "Generic (PLEG): container finished" podID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerID="0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d" exitCode=0 Oct 30 18:38:53.726999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.726951 6966 generic.go:356] "Generic (PLEG): container finished" podID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerID="6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4" exitCode=0 Oct 30 18:38:53.727174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.726996 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerDied","Data":"0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d"} Oct 30 18:38:53.727174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:53.727050 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerDied","Data":"6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4"} Oct 30 18:38:54.032219 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.032188 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:54.127581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127559 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.127698 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127659 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.127698 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127692 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.127879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127720 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.127879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127752 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.127879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127819 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw\" (UniqueName: \"kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.127879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127850 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.128089 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128066 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "gateway-rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.128151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128085 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-serving-cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.128270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128213 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.128270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.127912 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.128703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128281 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants\") pod \"8aae657c-c800-4102-ba42-9aba4f1113c3\" (UID: \"8aae657c-c800-4102-ba42-9aba4f1113c3\") " Oct 30 18:38:54.128703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128575 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:54.128703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128582 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-serving-cabundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-serving-cabundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.128703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128605 6966 reconciler_common.go:299] "Volume detached for volume \"gateway-rbac\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.128703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.128621 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.130300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.130269 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-gateway-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:54.130527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.130493 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "kube-api-access-czkmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:54.130527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.130505 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "tempo-data-science-tempomonolithic-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:54.130986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.130962 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "gateway-tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:54.131283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.131261 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55" pod "8aae657c-c800-4102-ba42-9aba4f1113c3" (UID: "8aae657c-c800-4102-ba42-9aba4f1113c3"). InnerVolumeSpecName "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:38:54.229687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.229660 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-mtls\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.229687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.229682 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czkmw\" (UniqueName: \"kubernetes.io/projected/8aae657c-c800-4102-ba42-9aba4f1113c3-kube-api-access-czkmw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.229799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.229694 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempomonolithic-gateway-serving-cert\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-data-science-tempomonolithic-gateway-serving-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.229799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.229715 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/8aae657c-c800-4102-ba42-9aba4f1113c3-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.229799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.229725 6966 reconciler_common.go:299] "Volume detached for volume \"gateway-tenants\" (UniqueName: \"kubernetes.io/secret/8aae657c-c800-4102-ba42-9aba4f1113c3-gateway-tenants\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:38:54.229799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.229748 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:38:54.237870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.237854 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:38:54.330897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.330872 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-88c8a23a-436d-4b38-8255-c71b8af6ca1c\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0e50aa39279f5db55\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-f1d243cd1eff0987a64de94f5fb75fb003a2545b22342d982b5bcfa8f4a85c68\"" Oct 30 18:38:54.735873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.735794 6966 generic.go:356] "Generic (PLEG): container finished" podID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerID="d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f" exitCode=0 Oct 30 18:38:54.735873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.735852 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerDied","Data":"d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f"} Oct 30 18:38:54.736081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.735908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" event={"ID":"8aae657c-c800-4102-ba42-9aba4f1113c3","Type":"ContainerDied","Data":"3cafe95dbb6040991097d5f29d2eeb8d63a0a8ba4f52e4af7a9e837103bd7ca7"} Oct 30 18:38:54.736081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.735854 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempomonolithic-0" Oct 30 18:38:54.736081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.735938 6966 scope.go:122] "RemoveContainer" containerID="0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d" Oct 30 18:38:54.745708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.745690 6966 scope.go:122] "RemoveContainer" containerID="d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f" Oct 30 18:38:54.755001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.754973 6966 scope.go:122] "RemoveContainer" containerID="6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4" Oct 30 18:38:54.762999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.762982 6966 scope.go:122] "RemoveContainer" containerID="0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d" Oct 30 18:38:54.763275 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:54.763248 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d\": container with ID starting with 0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d not found: ID does not exist" containerID="0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d" Oct 30 18:38:54.763370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.763273 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d"} err="failed to get container status \"0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d\": rpc error: code = NotFound desc = could not find container \"0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d\": container with ID starting with 0a2c08666f827615cb11d7f403ec490ee28bbe410d43ca2847401a11e9e29a4d not found: ID does not exist" Oct 30 18:38:54.763370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.763290 6966 scope.go:122] "RemoveContainer" containerID="d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f" Oct 30 18:38:54.763578 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:54.763562 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f\": container with ID starting with d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f not found: ID does not exist" containerID="d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f" Oct 30 18:38:54.763628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.763582 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f"} err="failed to get container status \"d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f\": rpc error: code = NotFound desc = could not find container \"d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f\": container with ID starting with d412a9c673e2c33d34047c78f9c205761395c8d2a8c2ef53558a35bcb025385f not found: ID does not exist" Oct 30 18:38:54.763628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.763601 6966 scope.go:122] "RemoveContainer" containerID="6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4" Oct 30 18:38:54.763851 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:38:54.763833 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4\": container with ID starting with 6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4 not found: ID does not exist" containerID="6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4" Oct 30 18:38:54.763894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.763856 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4"} err="failed to get container status \"6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4\": rpc error: code = NotFound desc = could not find container \"6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4\": container with ID starting with 6888b96294c558c02c0ba7f3ca2ea8c386250e4d788ee0b161b32c7c0045bec4 not found: ID does not exist" Oct 30 18:38:54.797772 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.797752 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:38:54.808655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:54.808633 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempomonolithic-0"] Oct 30 18:38:55.410769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.410737 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" path="/var/lib/kubelet/pods/8aae657c-c800-4102-ba42-9aba4f1113c3/volumes" Oct 30 18:38:55.723671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723595 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9rp52"] Oct 30 18:38:55.723928 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723916 6966 cpu_manager.go:414] "containerMap: removing container" podUID="776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" containerName="korrel8r" Oct 30 18:38:55.723976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723930 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" containerName="korrel8r" Oct 30 18:38:55.723976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723938 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway" Oct 30 18:38:55.723976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723943 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway" Oct 30 18:38:55.723976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723962 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8a3006e4-be5e-4af6-a6fe-0e79b2630f82" containerName="otc-container" Oct 30 18:38:55.723976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723967 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3006e4-be5e-4af6-a6fe-0e79b2630f82" containerName="otc-container" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723982 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway-opa" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723987 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway-opa" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.723997 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.724001 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.724055 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3006e4-be5e-4af6-a6fe-0e79b2630f82" containerName="otc-container" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.724065 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway-opa" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.724072 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.724079 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="776d6ee5-e229-4a6b-8fc8-850f0e85bb1e" containerName="korrel8r" Oct 30 18:38:55.724132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.724084 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aae657c-c800-4102-ba42-9aba4f1113c3" containerName="tempo-gateway" Oct 30 18:38:55.728850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.728829 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.753941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.753918 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:38:55.754069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.753948 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:38:55.754069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.753972 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:38:55.754069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.753945 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-5mt5g\"" Oct 30 18:38:55.754069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.754023 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:38:55.755656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.755640 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9rp52"] Oct 30 18:38:55.779913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.779890 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:55.842271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.842245 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.842271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.842271 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.842412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.842297 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8kfw\" (UniqueName: \"kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.942760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.942731 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.942760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.942760 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.942944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.942786 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8kfw\" (UniqueName: \"kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.943525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.943505 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.945094 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.945074 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.967880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.967858 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8kfw\" (UniqueName: \"kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw\") pod \"korrel8r-6644769df5-9rp52\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:55.988497 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:55.988419 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.040725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.040705 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:56.192762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.192723 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.341098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.341064 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9rp52"] Oct 30 18:38:56.346401 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:38:56.346314 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac84dea6_7562_4fc3_a2c7_9bff0d28766b.slice/crio-633b1fc2ac9fb90cc1dc8dbea3150d99cd096b035f6692a198c28b8c99a635bf WatchSource:0}: Error finding container 633b1fc2ac9fb90cc1dc8dbea3150d99cd096b035f6692a198c28b8c99a635bf: Status 404 returned error can't find the container with id 633b1fc2ac9fb90cc1dc8dbea3150d99cd096b035f6692a198c28b8c99a635bf Oct 30 18:38:56.371359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.371332 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:38:56.376524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.376507 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.396771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.396738 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:38:56.459347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.459308 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:38:56.462567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.462518 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:38:56.462567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.462544 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:38:56.462731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.462582 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-96qc2\"" Oct 30 18:38:56.465184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.465157 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.548281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.548252 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkmw\" (UniqueName: \"kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.548435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.548298 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.548536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.548453 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.548536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.548514 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.649490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.649401 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.649490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.649440 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.649490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.649485 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkmw\" (UniqueName: \"kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.649751 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.649514 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.650180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.650156 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.651800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.651780 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.651882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.651834 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.671331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.671301 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkmw\" (UniqueName: \"kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw\") pod \"data-science-collector-collector-1\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.679859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.679833 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:56.691568 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.691551 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:38:56.754684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.754657 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" event={"ID":"ac84dea6-7562-4fc3-a2c7-9bff0d28766b","Type":"ContainerStarted","Data":"2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755"} Oct 30 18:38:56.754826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.754710 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" event={"ID":"ac84dea6-7562-4fc3-a2c7-9bff0d28766b","Type":"ContainerStarted","Data":"633b1fc2ac9fb90cc1dc8dbea3150d99cd096b035f6692a198c28b8c99a635bf"} Oct 30 18:38:56.754887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.754851 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:38:56.800058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.800012 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" podStartSLOduration=1.800000346 podStartE2EDuration="1.800000346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:56.799545721 +0000 UTC m=+2499.505731078" watchObservedRunningTime="2026-10-30 18:38:56.800000346 +0000 UTC m=+2499.506185703" Oct 30 18:38:56.902238 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:56.902165 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:57.000956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.000926 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:38:57.006015 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:38:57.005985 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13baa04_1804_4c50_ae13_4a434beb04a6.slice/crio-6ffcbf13cf9d0aa51051d2d433d285abcae2072bcc149a1160fd875bd7062355 WatchSource:0}: Error finding container 6ffcbf13cf9d0aa51051d2d433d285abcae2072bcc149a1160fd875bd7062355: Status 404 returned error can't find the container with id 6ffcbf13cf9d0aa51051d2d433d285abcae2072bcc149a1160fd875bd7062355 Oct 30 18:38:57.008192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.008163 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:38:57.008277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.008218 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:38:57.171007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.170932 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:57.390268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.390238 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:38:57.761209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.761176 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f13baa04-1804-4c50-ae13-4a434beb04a6","Type":"ContainerStarted","Data":"6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7"} Oct 30 18:38:57.761621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.761232 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f13baa04-1804-4c50-ae13-4a434beb04a6","Type":"ContainerStarted","Data":"6ffcbf13cf9d0aa51051d2d433d285abcae2072bcc149a1160fd875bd7062355"} Oct 30 18:38:57.802074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:38:57.802029 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.802015379 podStartE2EDuration="1.802015379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:57.801624525 +0000 UTC m=+2500.507809882" watchObservedRunningTime="2026-10-30 18:38:57.802015379 +0000 UTC m=+2500.508200736" Oct 30 18:39:07.763246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:07.763207 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:39:10.894068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:10.894033 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9rp52"] Oct 30 18:39:10.894503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:10.894219 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" podUID="ac84dea6-7562-4fc3-a2c7-9bff0d28766b" containerName="korrel8r" containerID="cri-o://2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755" gracePeriod=30 Oct 30 18:39:11.137808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.137784 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:39:11.158022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.157976 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls\" (UniqueName: \"kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls\") pod \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " Oct 30 18:39:11.158022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.158022 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw\" (UniqueName: \"kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw\") pod \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " Oct 30 18:39:11.158197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.158078 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config\" (UniqueName: \"kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config\") pod \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\" (UID: \"ac84dea6-7562-4fc3-a2c7-9bff0d28766b\") " Oct 30 18:39:11.158569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.158539 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config" pod "ac84dea6-7562-4fc3-a2c7-9bff0d28766b" (UID: "ac84dea6-7562-4fc3-a2c7-9bff0d28766b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:11.159942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.159917 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls" pod "ac84dea6-7562-4fc3-a2c7-9bff0d28766b" (UID: "ac84dea6-7562-4fc3-a2c7-9bff0d28766b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:11.160116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.160093 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw" pod "ac84dea6-7562-4fc3-a2c7-9bff0d28766b" (UID: "ac84dea6-7562-4fc3-a2c7-9bff0d28766b"). InnerVolumeSpecName "kube-api-access-x8kfw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:11.258877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.258849 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.258877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.258874 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.259127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.258884 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8kfw\" (UniqueName: \"kubernetes.io/projected/ac84dea6-7562-4fc3-a2c7-9bff0d28766b-kube-api-access-x8kfw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.446733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.446705 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:39:11.446903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.446884 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="f13baa04-1804-4c50-ae13-4a434beb04a6" containerName="otc-container" containerID="cri-o://6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7" gracePeriod=30 Oct 30 18:39:11.651304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.651269 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.674852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.674805 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:11.764362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.764337 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw\" (UniqueName: \"kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw\") pod \"f13baa04-1804-4c50-ae13-4a434beb04a6\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " Oct 30 18:39:11.764529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.764429 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs\") pod \"f13baa04-1804-4c50-ae13-4a434beb04a6\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " Oct 30 18:39:11.764529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.764499 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal\") pod \"f13baa04-1804-4c50-ae13-4a434beb04a6\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " Oct 30 18:39:11.764654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.764528 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs\") pod \"f13baa04-1804-4c50-ae13-4a434beb04a6\" (UID: \"f13baa04-1804-4c50-ae13-4a434beb04a6\") " Oct 30 18:39:11.764910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.764880 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal" pod "f13baa04-1804-4c50-ae13-4a434beb04a6" (UID: "f13baa04-1804-4c50-ae13-4a434beb04a6"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:11.766617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.766590 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw" pod "f13baa04-1804-4c50-ae13-4a434beb04a6" (UID: "f13baa04-1804-4c50-ae13-4a434beb04a6"). InnerVolumeSpecName "kube-api-access-jtkmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:11.766617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.766590 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs" pod "f13baa04-1804-4c50-ae13-4a434beb04a6" (UID: "f13baa04-1804-4c50-ae13-4a434beb04a6"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:11.766791 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.766633 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs" pod "f13baa04-1804-4c50-ae13-4a434beb04a6" (UID: "f13baa04-1804-4c50-ae13-4a434beb04a6"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:11.830072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.830043 6966 generic.go:356] "Generic (PLEG): container finished" podID="f13baa04-1804-4c50-ae13-4a434beb04a6" containerID="6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7" exitCode=0 Oct 30 18:39:11.830215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.830105 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:11.830215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.830079 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f13baa04-1804-4c50-ae13-4a434beb04a6","Type":"ContainerDied","Data":"6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7"} Oct 30 18:39:11.830215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.830202 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"f13baa04-1804-4c50-ae13-4a434beb04a6","Type":"ContainerDied","Data":"6ffcbf13cf9d0aa51051d2d433d285abcae2072bcc149a1160fd875bd7062355"} Oct 30 18:39:11.830374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.830235 6966 scope.go:122] "RemoveContainer" containerID="6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7" Oct 30 18:39:11.831913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.831890 6966 generic.go:356] "Generic (PLEG): container finished" podID="ac84dea6-7562-4fc3-a2c7-9bff0d28766b" containerID="2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755" exitCode=0 Oct 30 18:39:11.832020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.831926 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" Oct 30 18:39:11.832020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.831926 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" event={"ID":"ac84dea6-7562-4fc3-a2c7-9bff0d28766b","Type":"ContainerDied","Data":"2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755"} Oct 30 18:39:11.832137 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.832048 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9rp52" event={"ID":"ac84dea6-7562-4fc3-a2c7-9bff0d28766b","Type":"ContainerDied","Data":"633b1fc2ac9fb90cc1dc8dbea3150d99cd096b035f6692a198c28b8c99a635bf"} Oct 30 18:39:11.839672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.839614 6966 scope.go:122] "RemoveContainer" containerID="6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7" Oct 30 18:39:11.839896 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:39:11.839876 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7\": container with ID starting with 6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7 not found: ID does not exist" containerID="6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7" Oct 30 18:39:11.839966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.839905 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7"} err="failed to get container status \"6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7\": rpc error: code = NotFound desc = could not find container \"6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7\": container with ID starting with 6b31d73ab122c515d43f00b339059bda25a53d9d3f82409a6f5a64f97fd119c7 not found: ID does not exist" Oct 30 18:39:11.839966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.839925 6966 scope.go:122] "RemoveContainer" containerID="2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755" Oct 30 18:39:11.847741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.847714 6966 scope.go:122] "RemoveContainer" containerID="2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755" Oct 30 18:39:11.847979 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:39:11.847961 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755\": container with ID starting with 2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755 not found: ID does not exist" containerID="2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755" Oct 30 18:39:11.848042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.847984 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755"} err="failed to get container status \"2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755\": rpc error: code = NotFound desc = could not find container \"2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755\": container with ID starting with 2d2b9df122c9e12dc0e73e7a5d7dd0bf7cabc049ec4555418d64415db3060755 not found: ID does not exist" Oct 30 18:39:11.866273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.866248 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.866273 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.866272 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f13baa04-1804-4c50-ae13-4a434beb04a6-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.866501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.866281 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f13baa04-1804-4c50-ae13-4a434beb04a6-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.866501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.866434 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtkmw\" (UniqueName: \"kubernetes.io/projected/f13baa04-1804-4c50-ae13-4a434beb04a6-kube-api-access-jtkmw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:39:11.883254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.883228 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9rp52"] Oct 30 18:39:11.887127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.887105 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9rp52"] Oct 30 18:39:11.894198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.894172 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:11.921182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.921155 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:39:11.926603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:11.926550 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:39:12.128045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:12.128008 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:12.379279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:12.379246 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:13.413665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:13.413625 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac84dea6-7562-4fc3-a2c7-9bff0d28766b" path="/var/lib/kubelet/pods/ac84dea6-7562-4fc3-a2c7-9bff0d28766b/volumes" Oct 30 18:39:13.414043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:13.414026 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13baa04-1804-4c50-ae13-4a434beb04a6" path="/var/lib/kubelet/pods/f13baa04-1804-4c50-ae13-4a434beb04a6/volumes" Oct 30 18:39:14.072223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072192 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9nrhm"] Oct 30 18:39:14.072546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072534 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ac84dea6-7562-4fc3-a2c7-9bff0d28766b" containerName="korrel8r" Oct 30 18:39:14.072609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072547 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac84dea6-7562-4fc3-a2c7-9bff0d28766b" containerName="korrel8r" Oct 30 18:39:14.072609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072562 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f13baa04-1804-4c50-ae13-4a434beb04a6" containerName="otc-container" Oct 30 18:39:14.072609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072567 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13baa04-1804-4c50-ae13-4a434beb04a6" containerName="otc-container" Oct 30 18:39:14.072733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072630 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac84dea6-7562-4fc3-a2c7-9bff0d28766b" containerName="korrel8r" Oct 30 18:39:14.072733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.072639 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13baa04-1804-4c50-ae13-4a434beb04a6" containerName="otc-container" Oct 30 18:39:14.077648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.077618 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.097591 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.097571 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9nrhm"] Oct 30 18:39:14.107512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.107488 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-k9cx5\"" Oct 30 18:39:14.107512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.107505 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:39:14.107512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.107505 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:39:14.107721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.107536 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:39:14.107721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.107582 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:39:14.125849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.125819 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:14.185798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.185770 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z65g\" (UniqueName: \"kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.185917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.185814 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.185983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.185937 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.287205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.287174 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z65g\" (UniqueName: \"kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.287391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.287223 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.287495 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.287398 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.287972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.287948 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.289925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.289901 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.310849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.310822 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z65g\" (UniqueName: \"kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g\") pod \"korrel8r-6644769df5-9nrhm\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.344111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.344051 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:14.390768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.390746 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.690641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.690571 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9nrhm"] Oct 30 18:39:14.696593 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:39:14.696563 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bcc8215_54ea_4aa4_9fbc_6cb04cbe5ac4.slice/crio-eef7c2be532198367e0074f59171f3b960d92dab30dfe524645ff016ac5893a5 WatchSource:0}: Error finding container eef7c2be532198367e0074f59171f3b960d92dab30dfe524645ff016ac5893a5: Status 404 returned error can't find the container with id eef7c2be532198367e0074f59171f3b960d92dab30dfe524645ff016ac5893a5 Oct 30 18:39:14.746873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.746842 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:39:14.751736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.751717 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.778029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.778007 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:39:14.792782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.792760 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlq5h\" (UniqueName: \"kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.792888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.792798 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.792943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.792903 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.792982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.792943 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.843169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.843129 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-af329f19\"" Oct 30 18:39:14.845663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.845619 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:39:14.845663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.845622 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:39:14.845825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.845627 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-8qss4\"" Oct 30 18:39:14.847762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.847737 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:14.851490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.851455 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" event={"ID":"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4","Type":"ContainerStarted","Data":"e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438"} Oct 30 18:39:14.851572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.851516 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" event={"ID":"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4","Type":"ContainerStarted","Data":"eef7c2be532198367e0074f59171f3b960d92dab30dfe524645ff016ac5893a5"} Oct 30 18:39:14.851613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.851603 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:14.890019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.889978 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" podStartSLOduration=0.889966885 podStartE2EDuration="889.966885ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:14.889478054 +0000 UTC m=+2517.595663402" watchObservedRunningTime="2026-10-30 18:39:14.889966885 +0000 UTC m=+2517.596152242" Oct 30 18:39:14.893801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.893781 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.893887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.893828 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.893887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.893854 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlq5h\" (UniqueName: \"kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.894013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.893986 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.894490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.894452 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.896189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.896173 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.896243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.896223 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:14.915927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:14.915904 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlq5h\" (UniqueName: \"kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h\") pod \"data-science-collector-collector-1\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:15.061497 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.061452 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:15.068934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.068912 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:39:15.384626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.384563 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:39:15.389726 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:39:15.389688 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146af9bb_528b_400c_b77c_0f7ce19e523e.slice/crio-5f9dacfea3b0b076f67b1219b8456a12ccda3dd55defc4331bd8fa9afc5582a4 WatchSource:0}: Error finding container 5f9dacfea3b0b076f67b1219b8456a12ccda3dd55defc4331bd8fa9afc5582a4: Status 404 returned error can't find the container with id 5f9dacfea3b0b076f67b1219b8456a12ccda3dd55defc4331bd8fa9afc5582a4 Oct 30 18:39:15.392348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.392304 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:39:15.392482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.392377 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:39:15.714818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.714740 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:15.858483 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.858441 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"146af9bb-528b-400c-b77c-0f7ce19e523e","Type":"ContainerStarted","Data":"836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683"} Oct 30 18:39:15.858650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.858501 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"146af9bb-528b-400c-b77c-0f7ce19e523e","Type":"ContainerStarted","Data":"5f9dacfea3b0b076f67b1219b8456a12ccda3dd55defc4331bd8fa9afc5582a4"} Oct 30 18:39:15.898644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.898596 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.898578412 podStartE2EDuration="1.898578412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:39:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:39:15.898164736 +0000 UTC m=+2518.604350093" watchObservedRunningTime="2026-10-30 18:39:15.898578412 +0000 UTC m=+2518.604763771" Oct 30 18:39:15.924884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:15.924854 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:17.198798 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:39:17.198761 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:39:21.645000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:21.644968 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:22.649616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:22.649576 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:39:25.860361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:39:25.860336 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:39:47.198640 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:39:47.198606 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:40:06.058846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.058810 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9nrhm"] Oct 30 18:40:06.059573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.059036 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" podUID="2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" containerName="korrel8r" containerID="cri-o://e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438" gracePeriod=30 Oct 30 18:40:06.313729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.313704 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:40:06.384412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.384385 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g\" (UniqueName: \"kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g\") pod \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " Oct 30 18:40:06.384610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.384445 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config\" (UniqueName: \"kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config\") pod \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " Oct 30 18:40:06.384610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.384483 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls\" (UniqueName: \"kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls\") pod \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\" (UID: \"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4\") " Oct 30 18:40:06.384790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.384765 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config" pod "2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" (UID: "2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:06.386583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.386555 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls" pod "2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" (UID: "2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:06.386704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.386629 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g" pod "2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" (UID: "2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4"). InnerVolumeSpecName "kube-api-access-9z65g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:06.467781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.467743 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:40:06.468034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.468007 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="146af9bb-528b-400c-b77c-0f7ce19e523e" containerName="otc-container" containerID="cri-o://836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683" gracePeriod=30 Oct 30 18:40:06.485837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.485810 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9z65g\" (UniqueName: \"kubernetes.io/projected/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-kube-api-access-9z65g\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.485837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.485835 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.486053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.485845 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.606326 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.606295 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:06.698358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.698337 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:06.788035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.788006 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs\") pod \"146af9bb-528b-400c-b77c-0f7ce19e523e\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " Oct 30 18:40:06.788184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.788040 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs\") pod \"146af9bb-528b-400c-b77c-0f7ce19e523e\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " Oct 30 18:40:06.788184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.788123 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h\" (UniqueName: \"kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h\") pod \"146af9bb-528b-400c-b77c-0f7ce19e523e\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " Oct 30 18:40:06.788184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.788161 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal\") pod \"146af9bb-528b-400c-b77c-0f7ce19e523e\" (UID: \"146af9bb-528b-400c-b77c-0f7ce19e523e\") " Oct 30 18:40:06.788597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.788568 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal" pod "146af9bb-528b-400c-b77c-0f7ce19e523e" (UID: "146af9bb-528b-400c-b77c-0f7ce19e523e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:06.790053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.790024 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs" pod "146af9bb-528b-400c-b77c-0f7ce19e523e" (UID: "146af9bb-528b-400c-b77c-0f7ce19e523e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:06.790634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.790609 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs" pod "146af9bb-528b-400c-b77c-0f7ce19e523e" (UID: "146af9bb-528b-400c-b77c-0f7ce19e523e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:06.790634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.790622 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h" pod "146af9bb-528b-400c-b77c-0f7ce19e523e" (UID: "146af9bb-528b-400c-b77c-0f7ce19e523e"). InnerVolumeSpecName "kube-api-access-jlq5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:06.832605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.832576 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:06.888857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.888802 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlq5h\" (UniqueName: \"kubernetes.io/projected/146af9bb-528b-400c-b77c-0f7ce19e523e-kube-api-access-jlq5h\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.888857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.888825 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/146af9bb-528b-400c-b77c-0f7ce19e523e-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.888857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.888836 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:06.888857 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:06.888846 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/146af9bb-528b-400c-b77c-0f7ce19e523e-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:40:07.042760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.042732 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:07.104700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.104668 6966 generic.go:356] "Generic (PLEG): container finished" podID="2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" containerID="e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438" exitCode=0 Oct 30 18:40:07.105076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.104705 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" event={"ID":"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4","Type":"ContainerDied","Data":"e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438"} Oct 30 18:40:07.105076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.104731 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" Oct 30 18:40:07.105076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.104752 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-9nrhm" event={"ID":"2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4","Type":"ContainerDied","Data":"eef7c2be532198367e0074f59171f3b960d92dab30dfe524645ff016ac5893a5"} Oct 30 18:40:07.105076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.104778 6966 scope.go:122] "RemoveContainer" containerID="e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438" Oct 30 18:40:07.106289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.106262 6966 generic.go:356] "Generic (PLEG): container finished" podID="146af9bb-528b-400c-b77c-0f7ce19e523e" containerID="836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683" exitCode=0 Oct 30 18:40:07.106423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.106316 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:07.106423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.106349 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"146af9bb-528b-400c-b77c-0f7ce19e523e","Type":"ContainerDied","Data":"836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683"} Oct 30 18:40:07.106423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.106404 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"146af9bb-528b-400c-b77c-0f7ce19e523e","Type":"ContainerDied","Data":"5f9dacfea3b0b076f67b1219b8456a12ccda3dd55defc4331bd8fa9afc5582a4"} Oct 30 18:40:07.114279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.114258 6966 scope.go:122] "RemoveContainer" containerID="e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438" Oct 30 18:40:07.114557 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:40:07.114532 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438\": container with ID starting with e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438 not found: ID does not exist" containerID="e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438" Oct 30 18:40:07.114621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.114558 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438"} err="failed to get container status \"e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438\": rpc error: code = NotFound desc = could not find container \"e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438\": container with ID starting with e0e77db52be626cd915373c07b892aa71d15d9748e6caea247c3244a98a0e438 not found: ID does not exist" Oct 30 18:40:07.114621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.114578 6966 scope.go:122] "RemoveContainer" containerID="836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683" Oct 30 18:40:07.122235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.122218 6966 scope.go:122] "RemoveContainer" containerID="836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683" Oct 30 18:40:07.122514 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:40:07.122495 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683\": container with ID starting with 836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683 not found: ID does not exist" containerID="836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683" Oct 30 18:40:07.122567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.122522 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683"} err="failed to get container status \"836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683\": rpc error: code = NotFound desc = could not find container \"836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683\": container with ID starting with 836d83d1f2dd691ad76f08dfa53089107cea04b414e5f66816b566c9c683e683 not found: ID does not exist" Oct 30 18:40:07.154410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.154360 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9nrhm"] Oct 30 18:40:07.158535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.158517 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-9nrhm"] Oct 30 18:40:07.192382 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.192352 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:40:07.196585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.196567 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:40:07.255564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.255543 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:07.411954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.411896 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146af9bb-528b-400c-b77c-0f7ce19e523e" path="/var/lib/kubelet/pods/146af9bb-528b-400c-b77c-0f7ce19e523e/volumes" Oct 30 18:40:07.412286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:07.412274 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" path="/var/lib/kubelet/pods/2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4/volumes" Oct 30 18:40:10.358213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:10.358178 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:17.198830 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:40:17.198788 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:40:21.743811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:21.743770 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:22.796048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:22.796017 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:37.649579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.649543 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9"] Oct 30 18:40:37.650161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.650141 6966 cpu_manager.go:414] "containerMap: removing container" podUID="146af9bb-528b-400c-b77c-0f7ce19e523e" containerName="otc-container" Oct 30 18:40:37.650250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.650164 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146af9bb-528b-400c-b77c-0f7ce19e523e" containerName="otc-container" Oct 30 18:40:37.650250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.650183 6966 cpu_manager.go:414] "containerMap: removing container" podUID="2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" containerName="korrel8r" Oct 30 18:40:37.650250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.650191 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" containerName="korrel8r" Oct 30 18:40:37.650398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.650305 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="146af9bb-528b-400c-b77c-0f7ce19e523e" containerName="otc-container" Oct 30 18:40:37.650398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.650318 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bcc8215-54ea-4aa4-9fbc-6cb04cbe5ac4" containerName="korrel8r" Oct 30 18:40:37.654724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.654703 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.670918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.670900 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:40:37.682164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.682146 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-n7lzr\"" Oct 30 18:40:37.687109 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.687080 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:40:37.687233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.687114 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:40:37.687233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.687217 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:40:37.688529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.688509 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9"] Oct 30 18:40:37.707383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.707357 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:37.712251 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.712233 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4tkd\" (UniqueName: \"kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.712330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.712258 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.712330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.712301 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.813338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.813315 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4tkd\" (UniqueName: \"kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.813436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.813344 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.813436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.813386 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.814158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.814139 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.815701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.815678 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.837995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.837971 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4tkd\" (UniqueName: \"kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd\") pod \"korrel8r-5d45699dc4-x28w9\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:37.932417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.932361 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:37.967101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:37.967071 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:38.142516 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.142486 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.267131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.267064 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9"] Oct 30 18:40:38.273678 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:40:38.273256 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588381e5_afd5_46d7_8680_2e733ade1799.slice/crio-f2193f327f7b4e60e62b8d0aaa12dc77ab003960968c2f511da3a6a102bd0adc WatchSource:0}: Error finding container f2193f327f7b4e60e62b8d0aaa12dc77ab003960968c2f511da3a6a102bd0adc: Status 404 returned error can't find the container with id f2193f327f7b4e60e62b8d0aaa12dc77ab003960968c2f511da3a6a102bd0adc Oct 30 18:40:38.305659 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.305628 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:40:38.310487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.310444 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.331233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.331202 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:40:38.337606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.337572 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-b6s97\"" Oct 30 18:40:38.337720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.337604 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:40:38.340557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.340536 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:40:38.393345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.393315 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.400873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.400844 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:40:38.419080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.419058 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.419182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.419089 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2dlj\" (UniqueName: \"kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.419182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.419118 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.419311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.419282 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.520282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.520216 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.520282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.520278 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.520488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.520296 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2dlj\" (UniqueName: \"kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.520488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.520336 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.521037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.521015 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.522721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.522696 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.522804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.522769 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.542214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.542195 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2dlj\" (UniqueName: \"kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj\") pod \"data-science-collector-collector-1\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.640968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.640942 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:40:38.649006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.648971 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:38.936721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.936693 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:40:38.941754 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:40:38.941729 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9efadc61_0db7_4ca0_a956_17325bc4fe85.slice/crio-1defaad6772bf743377addf12edbf481f78d7e8eeac028a782315595c9e68714 WatchSource:0}: Error finding container 1defaad6772bf743377addf12edbf481f78d7e8eeac028a782315595c9e68714: Status 404 returned error can't find the container with id 1defaad6772bf743377addf12edbf481f78d7e8eeac028a782315595c9e68714 Oct 30 18:40:38.943842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.943806 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:40:38.943942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:38.943870 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:40:39.268420 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.268333 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" event={"ID":"588381e5-afd5-46d7-8680-2e733ade1799","Type":"ContainerStarted","Data":"f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632"} Oct 30 18:40:39.268599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.268523 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:40:39.268599 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.268567 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" event={"ID":"588381e5-afd5-46d7-8680-2e733ade1799","Type":"ContainerStarted","Data":"f2193f327f7b4e60e62b8d0aaa12dc77ab003960968c2f511da3a6a102bd0adc"} Oct 30 18:40:39.270067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.270044 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9efadc61-0db7-4ca0-a956-17325bc4fe85","Type":"ContainerStarted","Data":"15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0"} Oct 30 18:40:39.270184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.270079 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9efadc61-0db7-4ca0-a956-17325bc4fe85","Type":"ContainerStarted","Data":"1defaad6772bf743377addf12edbf481f78d7e8eeac028a782315595c9e68714"} Oct 30 18:40:39.306823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.306779 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" podStartSLOduration=2.306767723 podStartE2EDuration="2.306767723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:39.30583055 +0000 UTC m=+2602.012015907" watchObservedRunningTime="2026-10-30 18:40:39.306767723 +0000 UTC m=+2602.012953123" Oct 30 18:40:39.345613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:39.345555 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.345539361 podStartE2EDuration="1.345539361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:39.345213165 +0000 UTC m=+2602.051398538" watchObservedRunningTime="2026-10-30 18:40:39.345539361 +0000 UTC m=+2602.051724722" Oct 30 18:40:41.533303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:41.533255 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.753706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:41.753673 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:41.948556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:41.948527 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:42.153041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.153010 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:42.632710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.632675 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt"] Oct 30 18:40:42.636638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.636617 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.661373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661353 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 18:40:42.661373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661367 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 18:40:42.661746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661721 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:40:42.661746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661742 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:40:42.662728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661749 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-w2vq8\"" Oct 30 18:40:42.662728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661795 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 18:40:42.662728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.661803 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 18:40:42.663998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.663973 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt"] Oct 30 18:40:42.761477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761432 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.761643 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761483 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.761643 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761503 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fz7b\" (UniqueName: \"kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.761776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761642 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.761776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761704 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.761891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761785 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.761891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.761823 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862417 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862452 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862512 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862549 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862575 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fz7b\" (UniqueName: \"kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862612 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.862805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.862650 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.863255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.863227 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.863376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.863287 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.863376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.863349 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.864941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.864921 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.865388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.865368 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.865624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.865604 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.884775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.884723 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fz7b\" (UniqueName: \"kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b\") pod \"tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:42.951223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:42.951197 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:43.277082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:43.276969 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt"] Oct 30 18:40:43.281726 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:40:43.281693 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf06cbd16_24f3_48c6_b34b_230b28724f4e.slice/crio-2a230353d25da15be32e5ffc8e7e8b95d0918b2b49e9f32aeb117987f4222b29 WatchSource:0}: Error finding container 2a230353d25da15be32e5ffc8e7e8b95d0918b2b49e9f32aeb117987f4222b29: Status 404 returned error can't find the container with id 2a230353d25da15be32e5ffc8e7e8b95d0918b2b49e9f32aeb117987f4222b29 Oct 30 18:40:43.289761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:43.289733 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" event={"ID":"f06cbd16-24f3-48c6-b34b-230b28724f4e","Type":"ContainerStarted","Data":"2a230353d25da15be32e5ffc8e7e8b95d0918b2b49e9f32aeb117987f4222b29"} Oct 30 18:40:43.870074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:43.870041 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:44.068551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.068524 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:44.296843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.296809 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" event={"ID":"f06cbd16-24f3-48c6-b34b-230b28724f4e","Type":"ContainerStarted","Data":"153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990"} Oct 30 18:40:44.297018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.296869 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" event={"ID":"f06cbd16-24f3-48c6-b34b-230b28724f4e","Type":"ContainerStarted","Data":"ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb"} Oct 30 18:40:44.297018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.296905 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:44.298295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.298269 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.96:8083/ready\": dial tcp 10.134.0.96:8083: connect: connection refused" Oct 30 18:40:44.360578 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.360509 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" podStartSLOduration=2.360496778 podStartE2EDuration="2.360496778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:44.349311698 +0000 UTC m=+2607.055497055" watchObservedRunningTime="2026-10-30 18:40:44.360496778 +0000 UTC m=+2607.066682134" Oct 30 18:40:44.759097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:44.759016 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:40:45.302153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:45.302124 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:45.324230 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:45.324208 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:45.324554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:45.324540 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:40:47.199544 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:40:47.199505 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:40:50.276564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:40:50.276533 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:41:17.198909 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:17.198875 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:41:21.871549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:21.871514 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:22.943017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:22.942971 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:32.036193 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:32.036160 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:46.408373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:46.408342 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:46.623572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:46.623525 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:46.883569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:46.883540 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:46.883800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:46.883778 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="9efadc61-0db7-4ca0-a956-17325bc4fe85" containerName="otc-container" containerID="cri-o://15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0" gracePeriod=30 Oct 30 18:41:47.110839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.110805 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:47.130984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.130964 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.200493 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:47.200393 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:41:47.246614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.246587 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs\") pod \"9efadc61-0db7-4ca0-a956-17325bc4fe85\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " Oct 30 18:41:47.246765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.246622 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs\") pod \"9efadc61-0db7-4ca0-a956-17325bc4fe85\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " Oct 30 18:41:47.246765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.246668 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj\" (UniqueName: \"kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj\") pod \"9efadc61-0db7-4ca0-a956-17325bc4fe85\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " Oct 30 18:41:47.246765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.246739 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal\" (UniqueName: \"kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal\") pod \"9efadc61-0db7-4ca0-a956-17325bc4fe85\" (UID: \"9efadc61-0db7-4ca0-a956-17325bc4fe85\") " Oct 30 18:41:47.247127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.247099 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal" pod "9efadc61-0db7-4ca0-a956-17325bc4fe85" (UID: "9efadc61-0db7-4ca0-a956-17325bc4fe85"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:47.248747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.248717 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs" pod "9efadc61-0db7-4ca0-a956-17325bc4fe85" (UID: "9efadc61-0db7-4ca0-a956-17325bc4fe85"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:47.248747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.248731 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs" pod "9efadc61-0db7-4ca0-a956-17325bc4fe85" (UID: "9efadc61-0db7-4ca0-a956-17325bc4fe85"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:47.248933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.248816 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj" pod "9efadc61-0db7-4ca0-a956-17325bc4fe85" (UID: "9efadc61-0db7-4ca0-a956-17325bc4fe85"). InnerVolumeSpecName "kube-api-access-x2dlj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:47.322370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.322336 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:47.347736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.347712 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:47.347736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.347734 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/9efadc61-0db7-4ca0-a956-17325bc4fe85-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:47.347879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.347744 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2dlj\" (UniqueName: \"kubernetes.io/projected/9efadc61-0db7-4ca0-a956-17325bc4fe85-kube-api-access-x2dlj\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:47.347879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.347754 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/9efadc61-0db7-4ca0-a956-17325bc4fe85-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:47.604108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.604080 6966 generic.go:356] "Generic (PLEG): container finished" podID="9efadc61-0db7-4ca0-a956-17325bc4fe85" containerID="15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0" exitCode=0 Oct 30 18:41:47.604546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.604128 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.604546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.604171 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9efadc61-0db7-4ca0-a956-17325bc4fe85","Type":"ContainerDied","Data":"15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0"} Oct 30 18:41:47.604546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.604229 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"9efadc61-0db7-4ca0-a956-17325bc4fe85","Type":"ContainerDied","Data":"1defaad6772bf743377addf12edbf481f78d7e8eeac028a782315595c9e68714"} Oct 30 18:41:47.604546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.604262 6966 scope.go:122] "RemoveContainer" containerID="15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0" Oct 30 18:41:47.612929 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.612907 6966 scope.go:122] "RemoveContainer" containerID="15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0" Oct 30 18:41:47.613167 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:47.613149 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0\": container with ID starting with 15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0 not found: ID does not exist" containerID="15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0" Oct 30 18:41:47.613228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.613181 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0"} err="failed to get container status \"15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0\": rpc error: code = NotFound desc = could not find container \"15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0\": container with ID starting with 15efcf5a357a13979093510a5c5d3d7a994954fd9c12fd167c3c3cc010ecb1e0 not found: ID does not exist" Oct 30 18:41:47.652178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.652158 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:47.663437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.663417 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:47.698814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.698797 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:47.699185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.699158 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9efadc61-0db7-4ca0-a956-17325bc4fe85" containerName="otc-container" Oct 30 18:41:47.699297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.699200 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9efadc61-0db7-4ca0-a956-17325bc4fe85" containerName="otc-container" Oct 30 18:41:47.699297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.699286 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9efadc61-0db7-4ca0-a956-17325bc4fe85" containerName="otc-container" Oct 30 18:41:47.703833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.703814 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.717694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.717673 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-b6s97\"" Oct 30 18:41:47.717694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.717672 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:41:47.717824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.717731 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:41:47.717824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.717773 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:41:47.725982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.725962 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:47.851132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.851100 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.851132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.851137 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.851321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.851169 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55lcd\" (UniqueName: \"kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.851321 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.851263 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.952666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.952608 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.952666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.952665 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.952924 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.952690 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.952924 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.952723 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55lcd\" (UniqueName: \"kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.953374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.953345 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.954950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.954932 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.955105 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.955085 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:47.974723 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:47.974697 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55lcd\" (UniqueName: \"kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd\") pod \"data-science-collector-collector-1\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:48.016312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.016289 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:48.312127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.312102 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:48.316394 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:41:48.316373 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod916fc318_aa64_43ee_92d6_b25ed3fa1e17.slice/crio-7f9769c551c7d456586875cc3c638b2323f1ca1b6fd87af81ee76e362391fbe7 WatchSource:0}: Error finding container 7f9769c551c7d456586875cc3c638b2323f1ca1b6fd87af81ee76e362391fbe7: Status 404 returned error can't find the container with id 7f9769c551c7d456586875cc3c638b2323f1ca1b6fd87af81ee76e362391fbe7 Oct 30 18:41:48.318370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.318338 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:41:48.318449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.318396 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:41:48.611989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.611905 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"916fc318-aa64-43ee-92d6-b25ed3fa1e17","Type":"ContainerStarted","Data":"d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe"} Oct 30 18:41:48.611989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.611959 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"916fc318-aa64-43ee-92d6-b25ed3fa1e17","Type":"ContainerStarted","Data":"7f9769c551c7d456586875cc3c638b2323f1ca1b6fd87af81ee76e362391fbe7"} Oct 30 18:41:48.652445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:48.652396 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.652380216 podStartE2EDuration="1.652380216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:41:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:41:48.651962737 +0000 UTC m=+2671.358148106" watchObservedRunningTime="2026-10-30 18:41:48.652380216 +0000 UTC m=+2671.358565576" Oct 30 18:41:49.411005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:49.410974 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9efadc61-0db7-4ca0-a956-17325bc4fe85" path="/var/lib/kubelet/pods/9efadc61-0db7-4ca0-a956-17325bc4fe85/volumes" Oct 30 18:41:50.794054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:50.794018 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9"] Oct 30 18:41:50.794510 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:50.794262 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" podUID="588381e5-afd5-46d7-8680-2e733ade1799" containerName="korrel8r" containerID="cri-o://f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632" gracePeriod=30 Oct 30 18:41:51.047059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.047013 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:41:51.160017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.159982 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:51.160281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.160253 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="916fc318-aa64-43ee-92d6-b25ed3fa1e17" containerName="otc-container" containerID="cri-o://d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe" gracePeriod=30 Oct 30 18:41:51.177082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.177057 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd\" (UniqueName: \"kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd\") pod \"588381e5-afd5-46d7-8680-2e733ade1799\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " Oct 30 18:41:51.177209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.177189 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls\" (UniqueName: \"kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls\") pod \"588381e5-afd5-46d7-8680-2e733ade1799\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " Oct 30 18:41:51.177274 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.177239 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config\" (UniqueName: \"kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config\") pod \"588381e5-afd5-46d7-8680-2e733ade1799\" (UID: \"588381e5-afd5-46d7-8680-2e733ade1799\") " Oct 30 18:41:51.177698 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.177639 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config" pod "588381e5-afd5-46d7-8680-2e733ade1799" (UID: "588381e5-afd5-46d7-8680-2e733ade1799"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:51.180983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.180959 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd" pod "588381e5-afd5-46d7-8680-2e733ade1799" (UID: "588381e5-afd5-46d7-8680-2e733ade1799"). InnerVolumeSpecName "kube-api-access-z4tkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:51.181072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.181054 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls" pod "588381e5-afd5-46d7-8680-2e733ade1799" (UID: "588381e5-afd5-46d7-8680-2e733ade1799"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:51.281582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.280438 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/588381e5-afd5-46d7-8680-2e733ade1799-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.281582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.280493 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/588381e5-afd5-46d7-8680-2e733ade1799-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.281582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.280510 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4tkd\" (UniqueName: \"kubernetes.io/projected/588381e5-afd5-46d7-8680-2e733ade1799-kube-api-access-z4tkd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.297672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.297594 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:51.392872 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.392847 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:51.481702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.481677 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs\") pod \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " Oct 30 18:41:51.481850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.481720 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs\") pod \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " Oct 30 18:41:51.481850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.481764 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd\" (UniqueName: \"kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd\") pod \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " Oct 30 18:41:51.481850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.481805 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal\" (UniqueName: \"kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal\") pod \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\" (UID: \"916fc318-aa64-43ee-92d6-b25ed3fa1e17\") " Oct 30 18:41:51.482250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.482219 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal" pod "916fc318-aa64-43ee-92d6-b25ed3fa1e17" (UID: "916fc318-aa64-43ee-92d6-b25ed3fa1e17"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:51.483881 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.483848 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs" pod "916fc318-aa64-43ee-92d6-b25ed3fa1e17" (UID: "916fc318-aa64-43ee-92d6-b25ed3fa1e17"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:51.483981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.483913 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs" pod "916fc318-aa64-43ee-92d6-b25ed3fa1e17" (UID: "916fc318-aa64-43ee-92d6-b25ed3fa1e17"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:51.483981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.483938 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd" pod "916fc318-aa64-43ee-92d6-b25ed3fa1e17" (UID: "916fc318-aa64-43ee-92d6-b25ed3fa1e17"). InnerVolumeSpecName "kube-api-access-55lcd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:51.517585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.517556 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:51.582788 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.582766 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.582788 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.582788 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.582917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.582797 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/916fc318-aa64-43ee-92d6-b25ed3fa1e17-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.582917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.582807 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55lcd\" (UniqueName: \"kubernetes.io/projected/916fc318-aa64-43ee-92d6-b25ed3fa1e17-kube-api-access-55lcd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:51.630613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.630581 6966 generic.go:356] "Generic (PLEG): container finished" podID="588381e5-afd5-46d7-8680-2e733ade1799" containerID="f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632" exitCode=0 Oct 30 18:41:51.630777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.630764 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" Oct 30 18:41:51.631210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.631187 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" event={"ID":"588381e5-afd5-46d7-8680-2e733ade1799","Type":"ContainerDied","Data":"f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632"} Oct 30 18:41:51.631345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.631236 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9" event={"ID":"588381e5-afd5-46d7-8680-2e733ade1799","Type":"ContainerDied","Data":"f2193f327f7b4e60e62b8d0aaa12dc77ab003960968c2f511da3a6a102bd0adc"} Oct 30 18:41:51.631345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.631263 6966 scope.go:122] "RemoveContainer" containerID="f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632" Oct 30 18:41:51.632882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.632746 6966 generic.go:356] "Generic (PLEG): container finished" podID="916fc318-aa64-43ee-92d6-b25ed3fa1e17" containerID="d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe" exitCode=0 Oct 30 18:41:51.632882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.632866 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"916fc318-aa64-43ee-92d6-b25ed3fa1e17","Type":"ContainerDied","Data":"d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe"} Oct 30 18:41:51.633010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.632908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"916fc318-aa64-43ee-92d6-b25ed3fa1e17","Type":"ContainerDied","Data":"7f9769c551c7d456586875cc3c638b2323f1ca1b6fd87af81ee76e362391fbe7"} Oct 30 18:41:51.633010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.632974 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:41:51.646676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.645542 6966 scope.go:122] "RemoveContainer" containerID="f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632" Oct 30 18:41:51.646676 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:51.645887 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632\": container with ID starting with f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632 not found: ID does not exist" containerID="f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632" Oct 30 18:41:51.646676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.645913 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632"} err="failed to get container status \"f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632\": rpc error: code = NotFound desc = could not find container \"f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632\": container with ID starting with f69dc29249c849588193119284eae3678e1e09475bec706c3347a07df79d2632 not found: ID does not exist" Oct 30 18:41:51.646676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.645931 6966 scope.go:122] "RemoveContainer" containerID="d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe" Oct 30 18:41:51.660065 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.660042 6966 scope.go:122] "RemoveContainer" containerID="d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe" Oct 30 18:41:51.660469 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:51.660435 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe\": container with ID starting with d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe not found: ID does not exist" containerID="d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe" Oct 30 18:41:51.660606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.660479 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe"} err="failed to get container status \"d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe\": rpc error: code = NotFound desc = could not find container \"d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe\": container with ID starting with d8772a3e0d9aca4f2edc222b2d938e8e2ff0452a5b6596caad19a6f41f0abebe not found: ID does not exist" Oct 30 18:41:51.684254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.684195 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9"] Oct 30 18:41:51.688240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.688196 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-x28w9"] Oct 30 18:41:51.738426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.738235 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:51.744993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.744966 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:41:51.763659 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.763633 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt"] Oct 30 18:41:51.764032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.763911 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway" containerID="cri-o://ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb" gracePeriod=30 Oct 30 18:41:51.764032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.763942 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway-opa" containerID="cri-o://153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990" gracePeriod=30 Oct 30 18:41:51.774700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:51.774676 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.071537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:52.071496 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:41:52.639327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:52.639291 6966 generic.go:356] "Generic (PLEG): container finished" podID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerID="153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990" exitCode=0 Oct 30 18:41:52.639522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:52.639360 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" event={"ID":"f06cbd16-24f3-48c6-b34b-230b28724f4e","Type":"ContainerDied","Data":"153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990"} Oct 30 18:41:53.010173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.010150 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:41:53.096638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096612 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096657 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b\" (UniqueName: \"kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096678 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096699 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096742 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096786 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.096818 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle\") pod \"f06cbd16-24f3-48c6-b34b-230b28724f4e\" (UID: \"f06cbd16-24f3-48c6-b34b-230b28724f4e\") " Oct 30 18:41:53.097400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.097097 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:53.097487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.097440 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:53.097487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.097449 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:53.098986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.098961 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "kube-api-access-7fz7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:53.099192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.099173 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:53.099389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.099369 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:53.102615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.102591 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant" pod "f06cbd16-24f3-48c6-b34b-230b28724f4e" (UID: "f06cbd16-24f3-48c6-b34b-230b28724f4e"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:53.198470 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198411 6966 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-serving-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.198470 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198431 6966 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.198470 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198440 6966 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-cabundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.198470 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198450 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.198648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198472 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fz7b\" (UniqueName: \"kubernetes.io/projected/f06cbd16-24f3-48c6-b34b-230b28724f4e-kube-api-access-7fz7b\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.198648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198486 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f06cbd16-24f3-48c6-b34b-230b28724f4e-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.198648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.198500 6966 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/f06cbd16-24f3-48c6-b34b-230b28724f4e-tenant\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:41:53.411044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.411017 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588381e5-afd5-46d7-8680-2e733ade1799" path="/var/lib/kubelet/pods/588381e5-afd5-46d7-8680-2e733ade1799/volumes" Oct 30 18:41:53.411387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.411375 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="916fc318-aa64-43ee-92d6-b25ed3fa1e17" path="/var/lib/kubelet/pods/916fc318-aa64-43ee-92d6-b25ed3fa1e17/volumes" Oct 30 18:41:53.648214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.648189 6966 generic.go:356] "Generic (PLEG): container finished" podID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerID="ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb" exitCode=0 Oct 30 18:41:53.648353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.648248 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" event={"ID":"f06cbd16-24f3-48c6-b34b-230b28724f4e","Type":"ContainerDied","Data":"ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb"} Oct 30 18:41:53.648353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.648268 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" Oct 30 18:41:53.648353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.648293 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt" event={"ID":"f06cbd16-24f3-48c6-b34b-230b28724f4e","Type":"ContainerDied","Data":"2a230353d25da15be32e5ffc8e7e8b95d0918b2b49e9f32aeb117987f4222b29"} Oct 30 18:41:53.648353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.648320 6966 scope.go:122] "RemoveContainer" containerID="153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990" Oct 30 18:41:53.656987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.656970 6966 scope.go:122] "RemoveContainer" containerID="ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb" Oct 30 18:41:53.664635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.664616 6966 scope.go:122] "RemoveContainer" containerID="153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990" Oct 30 18:41:53.664900 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:53.664879 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990\": container with ID starting with 153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990 not found: ID does not exist" containerID="153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990" Oct 30 18:41:53.664948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.664908 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990"} err="failed to get container status \"153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990\": rpc error: code = NotFound desc = could not find container \"153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990\": container with ID starting with 153e543fd163fa1f6472e8d640b14d5de25c811647182ddebb48852290c02990 not found: ID does not exist" Oct 30 18:41:53.664948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.664926 6966 scope.go:122] "RemoveContainer" containerID="ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb" Oct 30 18:41:53.665167 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:41:53.665149 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb\": container with ID starting with ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb not found: ID does not exist" containerID="ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb" Oct 30 18:41:53.665210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.665172 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb"} err="failed to get container status \"ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb\": rpc error: code = NotFound desc = could not find container \"ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb\": container with ID starting with ba4a2f4725e0f3abfc5d8c586c58706d26863b4ce2d69d5eeb4b1d54bcd6eefb not found: ID does not exist" Oct 30 18:41:53.699436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.699416 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt"] Oct 30 18:41:53.710533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:53.710510 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-54d6969f9b-kmtnt"] Oct 30 18:41:55.410949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:41:55.410911 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" path="/var/lib/kubelet/pods/f06cbd16-24f3-48c6-b34b-230b28724f4e/volumes" Oct 30 18:42:17.199009 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:42:17.198973 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:42:20.092262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:20.092225 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:22.006884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:22.006844 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:22.383693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:22.383662 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:23.029231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:23.029198 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:37.589339 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589247 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-p745v"] Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589594 6966 cpu_manager.go:414] "containerMap: removing container" podUID="588381e5-afd5-46d7-8680-2e733ade1799" containerName="korrel8r" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589605 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588381e5-afd5-46d7-8680-2e733ade1799" containerName="korrel8r" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589628 6966 cpu_manager.go:414] "containerMap: removing container" podUID="916fc318-aa64-43ee-92d6-b25ed3fa1e17" containerName="otc-container" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589633 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="916fc318-aa64-43ee-92d6-b25ed3fa1e17" containerName="otc-container" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589643 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway-opa" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589649 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway-opa" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589657 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589662 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway" Oct 30 18:42:37.589718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589721 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="916fc318-aa64-43ee-92d6-b25ed3fa1e17" containerName="otc-container" Oct 30 18:42:37.590025 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589730 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway-opa" Oct 30 18:42:37.590025 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589737 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f06cbd16-24f3-48c6-b34b-230b28724f4e" containerName="tempo-gateway" Oct 30 18:42:37.590025 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.589744 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="588381e5-afd5-46d7-8680-2e733ade1799" containerName="korrel8r" Oct 30 18:42:37.592882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.592863 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.618478 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.618441 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:42:37.618757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.618725 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:42:37.618757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.618748 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:42:37.618995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.618979 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:42:37.619090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.619063 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-hnxhh\"" Oct 30 18:42:37.620281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.620261 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-p745v"] Oct 30 18:42:37.626530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.626510 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg4tf\" (UniqueName: \"kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.626620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.626558 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.626700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.626683 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.640653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.640628 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:37.727636 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.727615 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.727731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.727669 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg4tf\" (UniqueName: \"kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.727731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.727707 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.728364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.728347 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.729908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.729892 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.750509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.750491 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg4tf\" (UniqueName: \"kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf\") pod \"korrel8r-5d45699dc4-p745v\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:37.905474 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:37.905390 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:38.200830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.200756 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-p745v"] Oct 30 18:42:38.205772 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:42:38.205740 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf49658_d474_4ce7_9635_9379fd9da1ee.slice/crio-c8bce7e1a6a358609dc5447aaf82aa3ea1312a3413ae7d9bef6f0839bb38f499 WatchSource:0}: Error finding container c8bce7e1a6a358609dc5447aaf82aa3ea1312a3413ae7d9bef6f0839bb38f499: Status 404 returned error can't find the container with id c8bce7e1a6a358609dc5447aaf82aa3ea1312a3413ae7d9bef6f0839bb38f499 Oct 30 18:42:38.252365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.252241 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:42:38.267678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.267080 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.278279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.278250 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:42:38.316207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.316183 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:42:38.316366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.316220 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:42:38.316366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.316290 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-qcbgc\"" Oct 30 18:42:38.332758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.332736 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.332874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.332777 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6sxq\" (UniqueName: \"kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.332874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.332844 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.332950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.332877 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.351267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.351248 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:42:38.434005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.433978 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.434130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.434019 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6sxq\" (UniqueName: \"kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.434130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.434049 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.434130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.434089 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.434713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.434689 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.436535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.436519 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.436699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.436679 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.456714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.456669 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6sxq\" (UniqueName: \"kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq\") pod \"data-science-collector-collector-1\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.580872 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.580846 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:42:38.861243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.861208 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" event={"ID":"ccf49658-d474-4ce7-9635-9379fd9da1ee","Type":"ContainerStarted","Data":"4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e"} Oct 30 18:42:38.861618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.861261 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" event={"ID":"ccf49658-d474-4ce7-9635-9379fd9da1ee","Type":"ContainerStarted","Data":"c8bce7e1a6a358609dc5447aaf82aa3ea1312a3413ae7d9bef6f0839bb38f499"} Oct 30 18:42:38.861618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.861307 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:42:38.877436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.877404 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:42:38.882260 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:42:38.882238 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383c6253_aba5_4c9e_b213_c9a9e8d3cf0e.slice/crio-d899159be5b9701eb4078c7de2995eb325f481b0e76037b2971d89e52d1c943c WatchSource:0}: Error finding container d899159be5b9701eb4078c7de2995eb325f481b0e76037b2971d89e52d1c943c: Status 404 returned error can't find the container with id d899159be5b9701eb4078c7de2995eb325f481b0e76037b2971d89e52d1c943c Oct 30 18:42:38.884192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.884154 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:42:38.884277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.884224 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:42:38.902186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:38.902145 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" podStartSLOduration=1.9021321690000002 podStartE2EDuration="1.902132169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:38.900631589 +0000 UTC m=+2721.606816947" watchObservedRunningTime="2026-10-30 18:42:38.902132169 +0000 UTC m=+2721.608317526" Oct 30 18:42:39.574325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:39.574295 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:39.868490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:39.868391 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e","Type":"ContainerStarted","Data":"93847268170c726ced52cf2be717a8ca8ab635853ca967bbce4751c0501c928b"} Oct 30 18:42:39.868490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:39.868445 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e","Type":"ContainerStarted","Data":"d899159be5b9701eb4078c7de2995eb325f481b0e76037b2971d89e52d1c943c"} Oct 30 18:42:39.908127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:39.908076 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.908062565 podStartE2EDuration="1.908062565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:39.907600182 +0000 UTC m=+2722.613785540" watchObservedRunningTime="2026-10-30 18:42:39.908062565 +0000 UTC m=+2722.614247948" Oct 30 18:42:40.111287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.111258 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.330338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.330306 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.537618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.537578 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.737890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.737862 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:42:40.965189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.965159 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8"] Oct 30 18:42:40.968932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.968910 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:40.983854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.983835 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-dockercfg-vf8g6\"" Oct 30 18:42:40.983994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.983979 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-distributor-mtls\"" Oct 30 18:42:40.995404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.995344 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack\"" Oct 30 18:42:40.995404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.995351 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-ca-bundle\"" Oct 30 18:42:40.996890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:40.996870 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8"] Oct 30 18:42:41.056830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.056803 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.056950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.056837 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfbl\" (UniqueName: \"kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.056950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.056864 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.057083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.057007 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.057143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.057108 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.059556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.059535 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg"] Oct 30 18:42:41.062952 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.062936 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.084102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.084081 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-query-frontend-mtls\"" Oct 30 18:42:41.084213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.084109 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"gcs-secret\"" Oct 30 18:42:41.086209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.086171 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg"] Oct 30 18:42:41.089180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.089160 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7"] Oct 30 18:42:41.092988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.092967 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.115375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.115351 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q"] Oct 30 18:42:41.119367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.119345 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7"] Oct 30 18:42:41.119494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.119444 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.140073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.140049 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q"] Oct 30 18:42:41.158067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158040 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.158163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158074 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.158163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158103 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.158163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158125 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.158317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158250 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.158371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158350 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.158407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158382 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.158445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158403 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.158445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158420 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.158445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158427 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.158561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158475 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.158561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158552 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctmxm\" (UniqueName: \"kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.158630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158573 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.158630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158606 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6b2q\" (UniqueName: \"kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.158705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158626 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.158705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158685 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.158776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158714 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t78h\" (UniqueName: \"kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.158776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158750 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.158851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158778 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.158851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158801 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.158851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158826 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.159005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.158985 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfbl\" (UniqueName: \"kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.159065 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.159027 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.159115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.159071 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.159663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.159637 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.159765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.159739 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.160966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.160939 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.181633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.181613 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfbl\" (UniqueName: \"kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl\") pod \"tempo-data-science-tempostack-distributor-856875765d-v8bl8\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.187490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.184723 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-querier-mtls\"" Oct 30 18:42:41.187490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.184825 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-compactor-mtls\"" Oct 30 18:42:41.195118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.194924 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt"] Oct 30 18:42:41.199701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.199685 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.216068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.216052 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:42:41.216243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.216228 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-cabundle\"" Oct 30 18:42:41.221621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.221600 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt"] Oct 30 18:42:41.260422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260355 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.260422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260394 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.260626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260421 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.260626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260452 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.260626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260491 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctmxm\" (UniqueName: \"kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.260626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260518 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.260626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260549 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.260849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260725 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6b2q\" (UniqueName: \"kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.260849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260764 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.260849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260813 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.260849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260836 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t78h\" (UniqueName: \"kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.261056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260861 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.261056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260891 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.261056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260934 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.261056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260936 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.261056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.260970 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.261320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261177 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.261320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261192 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.261320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261305 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.261320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261311 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.261564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261391 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.261564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261424 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75tb6\" (UniqueName: \"kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.261564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261499 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.261564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261539 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.261799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261564 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.261799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261595 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.261799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261625 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.261799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261660 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.261799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261693 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.262071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.261866 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.262071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.262039 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.262183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.262133 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.262183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.262164 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.262998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.262972 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.263413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.263394 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.263812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.263784 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.263965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.263946 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.264325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.264309 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.264961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.264938 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.265029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.264938 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.281054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.281035 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.282511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.282492 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6b2q\" (UniqueName: \"kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q\") pod \"tempo-data-science-tempostack-query-frontend-6674686c47-q55lg\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.282688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.282671 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t78h\" (UniqueName: \"kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h\") pod \"tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.283621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.283604 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctmxm\" (UniqueName: \"kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm\") pod \"tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.284108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.284083 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway\"" Oct 30 18:42:41.284108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.284085 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-dockercfg-vwtls\"" Oct 30 18:42:41.284257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.284163 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-tls\"" Oct 30 18:42:41.284257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.284187 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"tempo-data-science-tempostack-gateway-mtls\"" Oct 30 18:42:41.362848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.362821 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.362865 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.362985 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363080 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363136 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75tb6\" (UniqueName: \"kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363176 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363210 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363587 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.363708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363640 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.364001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.363982 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.365593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.365573 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.365686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.365622 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.366045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.366026 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.374871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.374853 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:41.386780 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.386755 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75tb6\" (UniqueName: \"kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6\") pod \"tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.406691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.406669 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:41.432888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.432857 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:41.514904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.514871 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:41.593119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.593077 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8"] Oct 30 18:42:41.598259 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:42:41.598226 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a8175e2_c5b0_4df1_9573_9f8140575c45.slice/crio-e3f55c6460593e14b4c8c7d6ad842295db0000ea03d6eaa95b350059b2508119 WatchSource:0}: Error finding container e3f55c6460593e14b4c8c7d6ad842295db0000ea03d6eaa95b350059b2508119: Status 404 returned error can't find the container with id e3f55c6460593e14b4c8c7d6ad842295db0000ea03d6eaa95b350059b2508119 Oct 30 18:42:41.680700 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.680679 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg"] Oct 30 18:42:41.685157 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:42:41.685134 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda45639af_531e_4b39_b369_560cda37b718.slice/crio-a76dfa0550cb2540da3965e568eb48cd7badbcd5fa0d0d2129cafd8fab0e87d5 WatchSource:0}: Error finding container a76dfa0550cb2540da3965e568eb48cd7badbcd5fa0d0d2129cafd8fab0e87d5: Status 404 returned error can't find the container with id a76dfa0550cb2540da3965e568eb48cd7badbcd5fa0d0d2129cafd8fab0e87d5 Oct 30 18:42:41.727032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.727010 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7"] Oct 30 18:42:41.758211 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.758189 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q"] Oct 30 18:42:41.817215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.817190 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt"] Oct 30 18:42:41.825760 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:42:41.825730 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode05e424c_453f_4e24_9957_4bf3e3f79799.slice/crio-5c2e5f825dc3c45ad4184595269f3d411901069a08b368aa79c92300054fba1a WatchSource:0}: Error finding container 5c2e5f825dc3c45ad4184595269f3d411901069a08b368aa79c92300054fba1a: Status 404 returned error can't find the container with id 5c2e5f825dc3c45ad4184595269f3d411901069a08b368aa79c92300054fba1a Oct 30 18:42:41.880870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.880842 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" event={"ID":"b9d49e16-289c-46e1-ab09-15b861f7421e","Type":"ContainerStarted","Data":"50c9ad36e65849581255b31f131b8949d350416a50e10e5e8c3790e4b5710ccf"} Oct 30 18:42:41.882353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.882326 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" event={"ID":"3a8175e2-c5b0-4df1-9573-9f8140575c45","Type":"ContainerStarted","Data":"36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58"} Oct 30 18:42:41.882501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.882384 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:42:41.882501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.882407 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" event={"ID":"3a8175e2-c5b0-4df1-9573-9f8140575c45","Type":"ContainerStarted","Data":"e3f55c6460593e14b4c8c7d6ad842295db0000ea03d6eaa95b350059b2508119"} Oct 30 18:42:41.883422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.883403 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" event={"ID":"e05e424c-453f-4e24-9957-4bf3e3f79799","Type":"ContainerStarted","Data":"5c2e5f825dc3c45ad4184595269f3d411901069a08b368aa79c92300054fba1a"} Oct 30 18:42:41.884597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.884578 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" event={"ID":"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe","Type":"ContainerStarted","Data":"5fd1b94401116450610624edb2614255c2aec83cdcaf32fcbf32d705984eeb64"} Oct 30 18:42:41.885733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.885712 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" event={"ID":"a45639af-531e-4b39-b369-560cda37b718","Type":"ContainerStarted","Data":"a76dfa0550cb2540da3965e568eb48cd7badbcd5fa0d0d2129cafd8fab0e87d5"} Oct 30 18:42:41.923773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:41.923592 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" podStartSLOduration=1.923564475 podStartE2EDuration="1.923564475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:41.921676989 +0000 UTC m=+2724.627862324" watchObservedRunningTime="2026-10-30 18:42:41.923564475 +0000 UTC m=+2724.629749834" Oct 30 18:42:42.893409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.893362 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" event={"ID":"b9d49e16-289c-46e1-ab09-15b861f7421e","Type":"ContainerStarted","Data":"aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc"} Oct 30 18:42:42.893905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.893441 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" event={"ID":"b9d49e16-289c-46e1-ab09-15b861f7421e","Type":"ContainerStarted","Data":"2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119"} Oct 30 18:42:42.893905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.893504 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:42.893905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.893535 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:42.894913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.894882 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway" probeResult="failure" output="Get \"https://10.134.0.104:8081/ready\": dial tcp 10.134.0.104:8081: connect: connection refused" Oct 30 18:42:42.895051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.894880 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway-opa" probeResult="failure" output="Get \"http://10.134.0.104:8083/ready\": dial tcp 10.134.0.104:8083: connect: connection refused" Oct 30 18:42:42.895208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.895188 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" event={"ID":"e05e424c-453f-4e24-9957-4bf3e3f79799","Type":"ContainerStarted","Data":"d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d"} Oct 30 18:42:42.895301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.895276 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:42:42.896714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.896695 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" event={"ID":"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe","Type":"ContainerStarted","Data":"37438b6f991e4267014d88fd8a2fe8c5870d7657f2cd816842928e71767cec75"} Oct 30 18:42:42.896794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.896732 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:42:42.898130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.898104 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" event={"ID":"a45639af-531e-4b39-b369-560cda37b718","Type":"ContainerStarted","Data":"b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb"} Oct 30 18:42:42.898268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.898255 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:42:42.962288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:42.962245 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" podStartSLOduration=1.962233072 podStartE2EDuration="1.962233072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:42.960076796 +0000 UTC m=+2725.666262153" watchObservedRunningTime="2026-10-30 18:42:42.962233072 +0000 UTC m=+2725.668418429" Oct 30 18:42:43.001744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:43.001705 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" podStartSLOduration=2.001692556 podStartE2EDuration="2.001692556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:43.000412691 +0000 UTC m=+2725.706598079" watchObservedRunningTime="2026-10-30 18:42:43.001692556 +0000 UTC m=+2725.707877913" Oct 30 18:42:43.043656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:43.043615 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podStartSLOduration=2.043602812 podStartE2EDuration="2.043602812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:43.042325346 +0000 UTC m=+2725.748510704" watchObservedRunningTime="2026-10-30 18:42:43.043602812 +0000 UTC m=+2725.749788168" Oct 30 18:42:43.084057 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:43.084014 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" podStartSLOduration=2.08400205 podStartE2EDuration="2.08400205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:42:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:42:43.082324785 +0000 UTC m=+2725.788510142" watchObservedRunningTime="2026-10-30 18:42:43.08400205 +0000 UTC m=+2725.790187406" Oct 30 18:42:43.904511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:43.904484 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:44.225330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:44.225255 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:42:47.199038 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:42:47.199003 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:42:49.871098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:42:49.871067 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:43:02.904114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:02.904077 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:43:03.909537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:03.909505 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:43:03.910057 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:03.909947 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:03.927131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:03.927103 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:43:13.908763 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:13.908719 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:17.198204 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:43:17.198170 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:43:22.149562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:22.149517 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:23.143180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:23.143140 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:23.907731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:23.907690 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:24.509301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:24.509253 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:33.907525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:33.907452 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:43:43.908304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:43.908270 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:43:46.589067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:46.589032 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:46.800116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:46.800050 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.025246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.025164 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:43:47.025495 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.025451 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" containerName="otc-container" containerID="cri-o://93847268170c726ced52cf2be717a8ca8ab635853ca967bbce4751c0501c928b" gracePeriod=30 Oct 30 18:43:47.201580 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:43:47.201544 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:43:47.228997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.228968 6966 generic.go:356] "Generic (PLEG): container finished" podID="383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" containerID="93847268170c726ced52cf2be717a8ca8ab635853ca967bbce4751c0501c928b" exitCode=0 Oct 30 18:43:47.229176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.229030 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e","Type":"ContainerDied","Data":"93847268170c726ced52cf2be717a8ca8ab635853ca967bbce4751c0501c928b"} Oct 30 18:43:47.251241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.251213 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.282545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.282481 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:47.411091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.411067 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal\") pod \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " Oct 30 18:43:47.411261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.411104 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs\") pod \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " Oct 30 18:43:47.411261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.411180 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs\") pod \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " Oct 30 18:43:47.411261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.411203 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq\" (UniqueName: \"kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq\") pod \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\" (UID: \"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e\") " Oct 30 18:43:47.411526 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.411496 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal" pod "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" (UID: "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:47.413405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.413370 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs" pod "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" (UID: "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:47.413540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.413406 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs" pod "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" (UID: "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:47.413540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.413400 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq" pod "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" (UID: "383c6253-aba5-4c9e-b213-c9a9e8d3cf0e"). InnerVolumeSpecName "kube-api-access-r6sxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:47.487984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.487959 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:43:47.511990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.511968 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:43:47.511990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.511989 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:43:47.512202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.511999 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:43:47.512202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:47.512007 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6sxq\" (UniqueName: \"kubernetes.io/projected/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e-kube-api-access-r6sxq\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:43:48.235515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.235483 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.235515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.235497 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"383c6253-aba5-4c9e-b213-c9a9e8d3cf0e","Type":"ContainerDied","Data":"d899159be5b9701eb4078c7de2995eb325f481b0e76037b2971d89e52d1c943c"} Oct 30 18:43:48.235978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.235559 6966 scope.go:122] "RemoveContainer" containerID="93847268170c726ced52cf2be717a8ca8ab635853ca967bbce4751c0501c928b" Oct 30 18:43:48.283627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.283605 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:43:48.294637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.294614 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:43:48.328936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.328917 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:43:48.329277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.329265 6966 cpu_manager.go:414] "containerMap: removing container" podUID="383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" containerName="otc-container" Oct 30 18:43:48.329388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.329278 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" containerName="otc-container" Oct 30 18:43:48.329388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.329369 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" containerName="otc-container" Oct 30 18:43:48.333939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.333917 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.347720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.347701 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-54f83bab\"" Oct 30 18:43:48.347816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.347757 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-qcbgc\"" Oct 30 18:43:48.347864 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.347825 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:43:48.347904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.347878 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:43:48.354267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.354243 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:43:48.420960 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.420934 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4r6v\" (UniqueName: \"kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.421091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.420985 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.421091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.421005 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.421168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.421111 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.521784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.521724 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.521784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.521768 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4r6v\" (UniqueName: \"kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.521930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.521845 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.521930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.521871 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.522562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.522544 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.524244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.524223 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.524383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.524363 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.544042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.544019 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4r6v\" (UniqueName: \"kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v\") pod \"data-science-collector-collector-1\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.645452 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.645422 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:43:48.943572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.943539 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:43:48.947847 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:43:48.947816 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3cb2a2_c4ff_4e34_b35a_0a969cb2fa90.slice/crio-de20c890ad493a7caa7045f650a89b672c95bbd970aea8b99f206b46bc22d028 WatchSource:0}: Error finding container de20c890ad493a7caa7045f650a89b672c95bbd970aea8b99f206b46bc22d028: Status 404 returned error can't find the container with id de20c890ad493a7caa7045f650a89b672c95bbd970aea8b99f206b46bc22d028 Oct 30 18:43:48.949984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.949951 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:43:48.950089 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:48.950018 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:43:49.243428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:49.243354 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90","Type":"ContainerStarted","Data":"e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e"} Oct 30 18:43:49.243428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:49.243401 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90","Type":"ContainerStarted","Data":"de20c890ad493a7caa7045f650a89b672c95bbd970aea8b99f206b46bc22d028"} Oct 30 18:43:49.284537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:49.284478 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.284446105 podStartE2EDuration="1.284446105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:43:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:43:49.283450884 +0000 UTC m=+2791.989636252" watchObservedRunningTime="2026-10-30 18:43:49.284446105 +0000 UTC m=+2791.990631464" Oct 30 18:43:49.411135 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:43:49.411104 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="383c6253-aba5-4c9e-b213-c9a9e8d3cf0e" path="/var/lib/kubelet/pods/383c6253-aba5-4c9e-b213-c9a9e8d3cf0e/volumes" Oct 30 18:44:17.199500 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:17.199409 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:44:20.103103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.103071 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-p745v"] Oct 30 18:44:20.103612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.103275 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" podUID="ccf49658-d474-4ce7-9635-9379fd9da1ee" containerName="korrel8r" containerID="cri-o://4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e" gracePeriod=30 Oct 30 18:44:20.360111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.360060 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:44:20.371325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.371307 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf\" (UniqueName: \"kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf\") pod \"ccf49658-d474-4ce7-9635-9379fd9da1ee\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " Oct 30 18:44:20.371422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.371395 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config\" (UniqueName: \"kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config\") pod \"ccf49658-d474-4ce7-9635-9379fd9da1ee\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " Oct 30 18:44:20.371511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.371494 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls\" (UniqueName: \"kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls\") pod \"ccf49658-d474-4ce7-9635-9379fd9da1ee\" (UID: \"ccf49658-d474-4ce7-9635-9379fd9da1ee\") " Oct 30 18:44:20.371673 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.371656 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config" pod "ccf49658-d474-4ce7-9635-9379fd9da1ee" (UID: "ccf49658-d474-4ce7-9635-9379fd9da1ee"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:20.371789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.371777 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ccf49658-d474-4ce7-9635-9379fd9da1ee-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.373335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.373313 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls" pod "ccf49658-d474-4ce7-9635-9379fd9da1ee" (UID: "ccf49658-d474-4ce7-9635-9379fd9da1ee"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:20.373518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.373499 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf" pod "ccf49658-d474-4ce7-9635-9379fd9da1ee" (UID: "ccf49658-d474-4ce7-9635-9379fd9da1ee"). InnerVolumeSpecName "kube-api-access-tg4tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:20.407948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.407921 6966 generic.go:356] "Generic (PLEG): container finished" podID="ccf49658-d474-4ce7-9635-9379fd9da1ee" containerID="4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e" exitCode=0 Oct 30 18:44:20.408069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.408003 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" event={"ID":"ccf49658-d474-4ce7-9635-9379fd9da1ee","Type":"ContainerDied","Data":"4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e"} Oct 30 18:44:20.408069 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.408048 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" event={"ID":"ccf49658-d474-4ce7-9635-9379fd9da1ee","Type":"ContainerDied","Data":"c8bce7e1a6a358609dc5447aaf82aa3ea1312a3413ae7d9bef6f0839bb38f499"} Oct 30 18:44:20.408183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.408079 6966 scope.go:122] "RemoveContainer" containerID="4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e" Oct 30 18:44:20.408247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.408230 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5d45699dc4-p745v" Oct 30 18:44:20.419361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.419343 6966 scope.go:122] "RemoveContainer" containerID="4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e" Oct 30 18:44:20.419674 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:20.419654 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e\": container with ID starting with 4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e not found: ID does not exist" containerID="4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e" Oct 30 18:44:20.419756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.419687 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e"} err="failed to get container status \"4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e\": rpc error: code = NotFound desc = could not find container \"4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e\": container with ID starting with 4a098ccbd8d202a419a809a27d9ba177c6190e883666ef58cf09c340c5d0e05e not found: ID does not exist" Oct 30 18:44:20.458022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.457995 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-p745v"] Oct 30 18:44:20.461782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.461761 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5d45699dc4-p745v"] Oct 30 18:44:20.472892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.472866 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg4tf\" (UniqueName: \"kubernetes.io/projected/ccf49658-d474-4ce7-9635-9379fd9da1ee-kube-api-access-tg4tf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.472995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.472896 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ccf49658-d474-4ce7-9635-9379fd9da1ee-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.476179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.476156 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:44:20.476397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.476373 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" containerName="otc-container" containerID="cri-o://e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e" gracePeriod=30 Oct 30 18:44:20.631131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.631044 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:20.716132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.716110 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:20.776082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.776057 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs\") pod \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " Oct 30 18:44:20.776248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.776103 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs\") pod \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " Oct 30 18:44:20.776248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.776239 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v\" (UniqueName: \"kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v\") pod \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " Oct 30 18:44:20.776371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.776318 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal\" (UniqueName: \"kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal\") pod \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\" (UID: \"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90\") " Oct 30 18:44:20.776739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.776709 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal" pod "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" (UID: "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:20.778367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.778339 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v" pod "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" (UID: "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90"). InnerVolumeSpecName "kube-api-access-q4r6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:20.778646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.778625 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs" pod "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" (UID: "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:20.779982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.779961 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs" pod "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" (UID: "7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:20.858418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.858389 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:20.877642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.877614 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.877642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.877637 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4r6v\" (UniqueName: \"kubernetes.io/projected/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-kube-api-access-q4r6v\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.877808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.877651 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:20.877808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:20.877665 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:21.078743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.078710 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg"] Oct 30 18:44:21.079227 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.079193 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" containerID="cri-o://b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb" gracePeriod=30 Oct 30 18:44:21.083934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.083904 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7"] Oct 30 18:44:21.084133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.084108 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" containerID="cri-o://d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d" gracePeriod=30 Oct 30 18:44:21.090091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.089748 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8"] Oct 30 18:44:21.090091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.089963 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" podUID="3a8175e2-c5b0-4df1-9573-9f8140575c45" containerName="tempo" containerID="cri-o://36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58" gracePeriod=30 Oct 30 18:44:21.093182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.092716 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt"] Oct 30 18:44:21.093182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.092937 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway" containerID="cri-o://2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119" gracePeriod=30 Oct 30 18:44:21.093182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.093065 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway-opa" containerID="cri-o://aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc" gracePeriod=30 Oct 30 18:44:21.095304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.095279 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q"] Oct 30 18:44:21.095499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.095477 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" containerID="cri-o://37438b6f991e4267014d88fd8a2fe8c5870d7657f2cd816842928e71767cec75" gracePeriod=30 Oct 30 18:44:21.130918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.129952 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:21.336792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.336705 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:21.412340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.412307 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf49658-d474-4ce7-9635-9379fd9da1ee" path="/var/lib/kubelet/pods/ccf49658-d474-4ce7-9635-9379fd9da1ee/volumes" Oct 30 18:44:21.414261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.414234 6966 generic.go:356] "Generic (PLEG): container finished" podID="7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" containerID="e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e" exitCode=0 Oct 30 18:44:21.414400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.414280 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:21.414400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.414308 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90","Type":"ContainerDied","Data":"e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e"} Oct 30 18:44:21.414400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.414351 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90","Type":"ContainerDied","Data":"de20c890ad493a7caa7045f650a89b672c95bbd970aea8b99f206b46bc22d028"} Oct 30 18:44:21.414400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.414373 6966 scope.go:122] "RemoveContainer" containerID="e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e" Oct 30 18:44:21.416715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.416683 6966 generic.go:356] "Generic (PLEG): container finished" podID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerID="aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc" exitCode=0 Oct 30 18:44:21.416854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.416757 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" event={"ID":"b9d49e16-289c-46e1-ab09-15b861f7421e","Type":"ContainerDied","Data":"aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc"} Oct 30 18:44:21.425088 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.425066 6966 scope.go:122] "RemoveContainer" containerID="e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e" Oct 30 18:44:21.425946 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:21.425916 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e\": container with ID starting with e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e not found: ID does not exist" containerID="e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e" Oct 30 18:44:21.426063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.425956 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e"} err="failed to get container status \"e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e\": rpc error: code = NotFound desc = could not find container \"e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e\": container with ID starting with e6b7ec45be75540e60d67368d3057dc522b60318a879e969eb39b6e2c530ab1e not found: ID does not exist" Oct 30 18:44:21.464623 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.464599 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:44:21.469075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:21.469053 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:44:22.231245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.231219 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:44:22.264091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.264062 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:22.291006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.290978 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf\") pod \"3a8175e2-c5b0-4df1-9573-9f8140575c45\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " Oct 30 18:44:22.291154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291054 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage\") pod \"3a8175e2-c5b0-4df1-9573-9f8140575c45\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " Oct 30 18:44:22.291154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291092 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl\" (UniqueName: \"kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl\") pod \"3a8175e2-c5b0-4df1-9573-9f8140575c45\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " Oct 30 18:44:22.291154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291130 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls\") pod \"3a8175e2-c5b0-4df1-9573-9f8140575c45\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " Oct 30 18:44:22.291342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291204 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle\") pod \"3a8175e2-c5b0-4df1-9573-9f8140575c45\" (UID: \"3a8175e2-c5b0-4df1-9573-9f8140575c45\") " Oct 30 18:44:22.291422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291389 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage" pod "3a8175e2-c5b0-4df1-9573-9f8140575c45" (UID: "3a8175e2-c5b0-4df1-9573-9f8140575c45"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:22.291499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291444 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf" pod "3a8175e2-c5b0-4df1-9573-9f8140575c45" (UID: "3a8175e2-c5b0-4df1-9573-9f8140575c45"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:22.291565 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291549 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.291627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291566 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-tmp-storage\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.291891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.291818 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle" pod "3a8175e2-c5b0-4df1-9573-9f8140575c45" (UID: "3a8175e2-c5b0-4df1-9573-9f8140575c45"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:22.293336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.293302 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls" pod "3a8175e2-c5b0-4df1-9573-9f8140575c45" (UID: "3a8175e2-c5b0-4df1-9573-9f8140575c45"). InnerVolumeSpecName "tempo-data-science-tempostack-distributor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.293562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.293540 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl" pod "3a8175e2-c5b0-4df1-9573-9f8140575c45" (UID: "3a8175e2-c5b0-4df1-9573-9f8140575c45"). InnerVolumeSpecName "kube-api-access-rhfbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:22.328396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.328377 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:44:22.391772 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.391744 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.391885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.391791 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.391885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.391847 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.391885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.391880 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.392058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.391917 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.392058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.391996 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.392058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392029 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6\" (UniqueName: \"kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6\") pod \"b9d49e16-289c-46e1-ab09-15b861f7421e\" (UID: \"b9d49e16-289c-46e1-ab09-15b861f7421e\") " Oct 30 18:44:22.392250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392227 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:22.392349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392323 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "cabundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:22.392400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392332 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.392400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392385 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfbl\" (UniqueName: \"kubernetes.io/projected/3a8175e2-c5b0-4df1-9573-9f8140575c45-kube-api-access-rhfbl\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.392538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392404 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-distributor-mtls\" (UniqueName: \"kubernetes.io/secret/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-distributor-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.392538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392421 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a8175e2-c5b0-4df1-9573-9f8140575c45-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.392538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.392428 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:22.394339 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.394315 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "kube-api-access-75tb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:22.394427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.394346 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "serving-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.394427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.394410 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "tempo-data-science-tempostack-gateway-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.399868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.399843 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant" pod "b9d49e16-289c-46e1-ab09-15b861f7421e" (UID: "b9d49e16-289c-46e1-ab09-15b861f7421e"). InnerVolumeSpecName "tenant". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:22.425364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.425339 6966 generic.go:356] "Generic (PLEG): container finished" podID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerID="2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119" exitCode=0 Oct 30 18:44:22.425455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.425384 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" Oct 30 18:44:22.425455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.425424 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" event={"ID":"b9d49e16-289c-46e1-ab09-15b861f7421e","Type":"ContainerDied","Data":"2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119"} Oct 30 18:44:22.425558 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.425495 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt" event={"ID":"b9d49e16-289c-46e1-ab09-15b861f7421e","Type":"ContainerDied","Data":"50c9ad36e65849581255b31f131b8949d350416a50e10e5e8c3790e4b5710ccf"} Oct 30 18:44:22.425558 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.425533 6966 scope.go:122] "RemoveContainer" containerID="aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc" Oct 30 18:44:22.426777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.426758 6966 generic.go:356] "Generic (PLEG): container finished" podID="3a8175e2-c5b0-4df1-9573-9f8140575c45" containerID="36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58" exitCode=0 Oct 30 18:44:22.426869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.426779 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" event={"ID":"3a8175e2-c5b0-4df1-9573-9f8140575c45","Type":"ContainerDied","Data":"36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58"} Oct 30 18:44:22.426869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.426807 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" Oct 30 18:44:22.426869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.426821 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8" event={"ID":"3a8175e2-c5b0-4df1-9573-9f8140575c45","Type":"ContainerDied","Data":"e3f55c6460593e14b4c8c7d6ad842295db0000ea03d6eaa95b350059b2508119"} Oct 30 18:44:22.433776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.433756 6966 scope.go:122] "RemoveContainer" containerID="2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119" Oct 30 18:44:22.442769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.442753 6966 scope.go:122] "RemoveContainer" containerID="aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc" Oct 30 18:44:22.443019 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:22.442997 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc\": container with ID starting with aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc not found: ID does not exist" containerID="aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc" Oct 30 18:44:22.443086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.443030 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc"} err="failed to get container status \"aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc\": rpc error: code = NotFound desc = could not find container \"aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc\": container with ID starting with aeacf01cd1bd16fb96e1405b8b892e682b32da60acaedb1e181fd1a95ad3c4cc not found: ID does not exist" Oct 30 18:44:22.443086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.443053 6966 scope.go:122] "RemoveContainer" containerID="2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119" Oct 30 18:44:22.443300 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:22.443284 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119\": container with ID starting with 2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119 not found: ID does not exist" containerID="2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119" Oct 30 18:44:22.443367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.443307 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119"} err="failed to get container status \"2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119\": rpc error: code = NotFound desc = could not find container \"2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119\": container with ID starting with 2e9c3cded94ea2fed16e618f747d0647d8d3b07c88a14f6468b902e64900c119 not found: ID does not exist" Oct 30 18:44:22.443367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.443326 6966 scope.go:122] "RemoveContainer" containerID="36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58" Oct 30 18:44:22.450988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.450970 6966 scope.go:122] "RemoveContainer" containerID="36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58" Oct 30 18:44:22.451226 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:22.451207 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58\": container with ID starting with 36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58 not found: ID does not exist" containerID="36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58" Oct 30 18:44:22.451276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.451231 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58"} err="failed to get container status \"36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58\": rpc error: code = NotFound desc = could not find container \"36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58\": container with ID starting with 36f57bcd94f9bfcfbcfc97c82d597244b49990a85d9f2ab73727e6b62492da58 not found: ID does not exist" Oct 30 18:44:22.473916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.473873 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt"] Oct 30 18:44:22.478290 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.478270 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-gateway-6b75bc4cf6-h9djt"] Oct 30 18:44:22.493246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.493224 6966 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.493353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.493247 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75tb6\" (UniqueName: \"kubernetes.io/projected/b9d49e16-289c-46e1-ab09-15b861f7421e-kube-api-access-75tb6\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.493353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.493262 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-gateway-mtls\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tempo-data-science-tempostack-gateway-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.493353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.493276 6966 reconciler_common.go:299] "Volume detached for volume \"serving-certs\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-serving-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.493353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.493290 6966 reconciler_common.go:299] "Volume detached for volume \"cabundle\" (UniqueName: \"kubernetes.io/configmap/b9d49e16-289c-46e1-ab09-15b861f7421e-cabundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.493353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.493301 6966 reconciler_common.go:299] "Volume detached for volume \"tenant\" (UniqueName: \"kubernetes.io/secret/b9d49e16-289c-46e1-ab09-15b861f7421e-tenant\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:22.509575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.509553 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8"] Oct 30 18:44:22.513500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:22.513482 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-distributor-856875765d-v8bl8"] Oct 30 18:44:23.260022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.259986 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:23.411366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.411299 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a8175e2-c5b0-4df1-9573-9f8140575c45" path="/var/lib/kubelet/pods/3a8175e2-c5b0-4df1-9573-9f8140575c45/volumes" Oct 30 18:44:23.411746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.411731 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" path="/var/lib/kubelet/pods/7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90/volumes" Oct 30 18:44:23.412121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.412108 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" path="/var/lib/kubelet/pods/b9d49e16-289c-46e1-ab09-15b861f7421e/volumes" Oct 30 18:44:23.908754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.908721 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:23.909451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.909426 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:23.950138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:23.950115 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:24.025688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.025651 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:24.443118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.443089 6966 generic.go:356] "Generic (PLEG): container finished" podID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerID="37438b6f991e4267014d88fd8a2fe8c5870d7657f2cd816842928e71767cec75" exitCode=0 Oct 30 18:44:24.443494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.443157 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" event={"ID":"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe","Type":"ContainerDied","Data":"37438b6f991e4267014d88fd8a2fe8c5870d7657f2cd816842928e71767cec75"} Oct 30 18:44:24.541539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.541516 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:44:24.612438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612380 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h\" (UniqueName: \"kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h\") pod \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " Oct 30 18:44:24.612438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612435 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle\") pod \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " Oct 30 18:44:24.612662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612481 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf\") pod \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " Oct 30 18:44:24.612662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612503 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage\") pod \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " Oct 30 18:44:24.612662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612608 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls\") pod \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " Oct 30 18:44:24.612662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612649 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key\") pod \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\" (UID: \"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe\") " Oct 30 18:44:24.612884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612855 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle" pod "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" (UID: "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:24.612884 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612863 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage" pod "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" (UID: "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:24.612992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612900 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf" pod "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" (UID: "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:24.612992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612978 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:24.613066 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.612998 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-tmp-storage\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:24.614504 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.614482 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h" pod "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" (UID: "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe"). InnerVolumeSpecName "kube-api-access-8t78h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:24.614693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.614667 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key" pod "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" (UID: "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:24.614693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.614687 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls" pod "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" (UID: "36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe"). InnerVolumeSpecName "tempo-data-science-tempostack-compactor-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:24.713425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.713401 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t78h\" (UniqueName: \"kubernetes.io/projected/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-kube-api-access-8t78h\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:24.713425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.713421 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:24.713577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.713431 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-compactor-mtls\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-tempo-data-science-tempostack-compactor-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:24.713577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.713440 6966 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe-storage-gcs-key\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:24.934448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934375 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-td6q2"] Oct 30 18:44:24.934759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934747 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway-opa" Oct 30 18:44:24.934810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934761 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway-opa" Oct 30 18:44:24.934810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934774 6966 cpu_manager.go:414] "containerMap: removing container" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" Oct 30 18:44:24.934810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934779 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" Oct 30 18:44:24.934810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934800 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3a8175e2-c5b0-4df1-9573-9f8140575c45" containerName="tempo" Oct 30 18:44:24.934810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934806 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a8175e2-c5b0-4df1-9573-9f8140575c45" containerName="tempo" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934820 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ccf49658-d474-4ce7-9635-9379fd9da1ee" containerName="korrel8r" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934826 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf49658-d474-4ce7-9635-9379fd9da1ee" containerName="korrel8r" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934835 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" containerName="otc-container" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934841 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" containerName="otc-container" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934854 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934858 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934918 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway-opa" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934927 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" containerName="tempo" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934933 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a8175e2-c5b0-4df1-9573-9f8140575c45" containerName="tempo" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934940 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf49658-d474-4ce7-9635-9379fd9da1ee" containerName="korrel8r" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934948 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b3cb2a2-c4ff-4e34-b35a-0a969cb2fa90" containerName="otc-container" Oct 30 18:44:24.934974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.934957 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9d49e16-289c-46e1-ab09-15b861f7421e" containerName="tempo-gateway" Oct 30 18:44:24.938395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.938356 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:24.952533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.952513 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:44:24.952625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.952540 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-5sxt7\"" Oct 30 18:44:24.952625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.952558 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:44:24.957965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.957946 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-td6q2"] Oct 30 18:44:24.977699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:24.977657 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:25.016436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.016415 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.016549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.016454 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.016612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.016596 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncj7b\" (UniqueName: \"kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.117982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.117957 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncj7b\" (UniqueName: \"kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.118083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.118024 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.118083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.118058 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.118889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.118861 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.120472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.120438 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.140245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.140222 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncj7b\" (UniqueName: \"kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b\") pod \"korrel8r-6644769df5-td6q2\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.185311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.185261 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:25.251284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.251264 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:25.452600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.452532 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" Oct 30 18:44:25.452600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.452562 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q" event={"ID":"36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe","Type":"ContainerDied","Data":"5fd1b94401116450610624edb2614255c2aec83cdcaf32fcbf32d705984eeb64"} Oct 30 18:44:25.453024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.452618 6966 scope.go:122] "RemoveContainer" containerID="37438b6f991e4267014d88fd8a2fe8c5870d7657f2cd816842928e71767cec75" Oct 30 18:44:25.501912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.501881 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q"] Oct 30 18:44:25.512196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.512166 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-compactor-649c6db4b8-jvt8q"] Oct 30 18:44:25.549698 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.549666 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-td6q2"] Oct 30 18:44:25.556125 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:44:25.556096 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef81d08_ae85_4d6d_940a_9f2375ee38a9.slice/crio-1732bdb7adca55a26f7e5961c7a9ef3b503c45d3374cc0ce6c66ad2901ce8aa7 WatchSource:0}: Error finding container 1732bdb7adca55a26f7e5961c7a9ef3b503c45d3374cc0ce6c66ad2901ce8aa7: Status 404 returned error can't find the container with id 1732bdb7adca55a26f7e5961c7a9ef3b503c45d3374cc0ce6c66ad2901ce8aa7 Oct 30 18:44:25.650394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.650364 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:25.853139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:25.853109 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:26.463627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:26.463591 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" event={"ID":"6ef81d08-ae85-4d6d-940a-9f2375ee38a9","Type":"ContainerStarted","Data":"f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493"} Oct 30 18:44:26.464127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:26.463651 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" event={"ID":"6ef81d08-ae85-4d6d-940a-9f2375ee38a9","Type":"ContainerStarted","Data":"1732bdb7adca55a26f7e5961c7a9ef3b503c45d3374cc0ce6c66ad2901ce8aa7"} Oct 30 18:44:26.464127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:26.463813 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:26.503371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:26.503325 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" podStartSLOduration=2.50331293 podStartE2EDuration="2.50331293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:26.502161791 +0000 UTC m=+2829.208347151" watchObservedRunningTime="2026-10-30 18:44:26.50331293 +0000 UTC m=+2829.209498287" Oct 30 18:44:26.576284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:26.576256 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:26.793051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:26.792971 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:27.411233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:27.411199 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe" path="/var/lib/kubelet/pods/36f406bb-72ad-40e3-ae17-bd0bdcaf1cbe/volumes" Oct 30 18:44:33.907431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:33.907384 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:33.908143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:33.908119 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:37.471663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:37.471632 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:39.980934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:39.980892 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:40.183503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:40.183444 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:42.828319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:42.828287 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-td6q2"] Oct 30 18:44:42.828753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:42.828482 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" podUID="6ef81d08-ae85-4d6d-940a-9f2375ee38a9" containerName="korrel8r" containerID="cri-o://f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493" gracePeriod=30 Oct 30 18:44:43.066531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.066506 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:43.162235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.162172 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b\" (UniqueName: \"kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b\") pod \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " Oct 30 18:44:43.162377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.162299 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls\" (UniqueName: \"kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls\") pod \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " Oct 30 18:44:43.162377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.162352 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config\" (UniqueName: \"kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config\") pod \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\" (UID: \"6ef81d08-ae85-4d6d-940a-9f2375ee38a9\") " Oct 30 18:44:43.162749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.162721 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config" pod "6ef81d08-ae85-4d6d-940a-9f2375ee38a9" (UID: "6ef81d08-ae85-4d6d-940a-9f2375ee38a9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:43.164294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.164269 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls" pod "6ef81d08-ae85-4d6d-940a-9f2375ee38a9" (UID: "6ef81d08-ae85-4d6d-940a-9f2375ee38a9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:43.164976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.164960 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b" pod "6ef81d08-ae85-4d6d-940a-9f2375ee38a9" (UID: "6ef81d08-ae85-4d6d-940a-9f2375ee38a9"). InnerVolumeSpecName "kube-api-access-ncj7b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:43.263316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.263284 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.263316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.263312 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.263316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.263321 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncj7b\" (UniqueName: \"kubernetes.io/projected/6ef81d08-ae85-4d6d-940a-9f2375ee38a9-kube-api-access-ncj7b\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:43.375993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.375963 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.547875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.547840 6966 generic.go:356] "Generic (PLEG): container finished" podID="6ef81d08-ae85-4d6d-940a-9f2375ee38a9" containerID="f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493" exitCode=0 Oct 30 18:44:43.548051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.547890 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" Oct 30 18:44:43.548051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.547896 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" event={"ID":"6ef81d08-ae85-4d6d-940a-9f2375ee38a9","Type":"ContainerDied","Data":"f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493"} Oct 30 18:44:43.548051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.547957 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-6644769df5-td6q2" event={"ID":"6ef81d08-ae85-4d6d-940a-9f2375ee38a9","Type":"ContainerDied","Data":"1732bdb7adca55a26f7e5961c7a9ef3b503c45d3374cc0ce6c66ad2901ce8aa7"} Oct 30 18:44:43.548051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.547978 6966 scope.go:122] "RemoveContainer" containerID="f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493" Oct 30 18:44:43.559001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.558984 6966 scope.go:122] "RemoveContainer" containerID="f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493" Oct 30 18:44:43.559311 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:43.559272 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493\": container with ID starting with f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493 not found: ID does not exist" containerID="f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493" Oct 30 18:44:43.559454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.559322 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493"} err="failed to get container status \"f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493\": rpc error: code = NotFound desc = could not find container \"f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493\": container with ID starting with f81ea081723b7b00b9e2cffa4f8e2003605f232084d136c09cf718ef81d00493 not found: ID does not exist" Oct 30 18:44:43.587844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.587809 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.596132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.596110 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-td6q2"] Oct 30 18:44:43.602436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.602415 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-6644769df5-td6q2"] Oct 30 18:44:43.813366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.813334 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:43.906880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.906837 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:43.907258 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.906957 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:44:43.910559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.910533 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:44:43.910678 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:43.910612 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:44:44.046488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:44.046447 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:45.411529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:45.411497 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef81d08-ae85-4d6d-940a-9f2375ee38a9" path="/var/lib/kubelet/pods/6ef81d08-ae85-4d6d-940a-9f2375ee38a9/volumes" Oct 30 18:44:47.199893 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:47.199853 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:44:51.269482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.269439 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:44:51.291059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.291032 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:44:51.431962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.431888 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage\") pod \"a45639af-531e-4b39-b369-560cda37b718\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " Oct 30 18:44:51.431962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.431926 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key\") pod \"a45639af-531e-4b39-b369-560cda37b718\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " Oct 30 18:44:51.431962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.431954 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle\") pod \"e05e424c-453f-4e24-9957-4bf3e3f79799\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " Oct 30 18:44:51.432245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432037 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage\") pod \"e05e424c-453f-4e24-9957-4bf3e3f79799\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " Oct 30 18:44:51.432245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432096 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q\" (UniqueName: \"kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q\") pod \"a45639af-531e-4b39-b369-560cda37b718\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " Oct 30 18:44:51.432245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432129 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls\") pod \"a45639af-531e-4b39-b369-560cda37b718\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " Oct 30 18:44:51.432245 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432164 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf\") pod \"a45639af-531e-4b39-b369-560cda37b718\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " Oct 30 18:44:51.432494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432268 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage" pod "a45639af-531e-4b39-b369-560cda37b718" (UID: "a45639af-531e-4b39-b369-560cda37b718"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:51.432494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432309 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls\") pod \"e05e424c-453f-4e24-9957-4bf3e3f79799\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " Oct 30 18:44:51.432494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432418 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle" pod "e05e424c-453f-4e24-9957-4bf3e3f79799" (UID: "e05e424c-453f-4e24-9957-4bf3e3f79799"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:51.432494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432450 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm\" (UniqueName: \"kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm\") pod \"e05e424c-453f-4e24-9957-4bf3e3f79799\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " Oct 30 18:44:51.432732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432534 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf\") pod \"e05e424c-453f-4e24-9957-4bf3e3f79799\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " Oct 30 18:44:51.432732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432530 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage" pod "e05e424c-453f-4e24-9957-4bf3e3f79799" (UID: "e05e424c-453f-4e24-9957-4bf3e3f79799"). InnerVolumeSpecName "tempo-tmp-storage". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:44:51.432732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432599 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle\") pod \"a45639af-531e-4b39-b369-560cda37b718\" (UID: \"a45639af-531e-4b39-b369-560cda37b718\") " Oct 30 18:44:51.432732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432635 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key\") pod \"e05e424c-453f-4e24-9957-4bf3e3f79799\" (UID: \"e05e424c-453f-4e24-9957-4bf3e3f79799\") " Oct 30 18:44:51.432956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432855 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/a45639af-531e-4b39-b369-560cda37b718-tempo-tmp-storage\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.432956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432875 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.432956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432890 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-tmp-storage\" (UniqueName: \"kubernetes.io/empty-dir/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-tmp-storage\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.433120 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.432969 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf" pod "e05e424c-453f-4e24-9957-4bf3e3f79799" (UID: "e05e424c-453f-4e24-9957-4bf3e3f79799"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:51.433177 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.433116 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf" pod "a45639af-531e-4b39-b369-560cda37b718" (UID: "a45639af-531e-4b39-b369-560cda37b718"). InnerVolumeSpecName "tempo-conf". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:51.433480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.433435 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle" pod "a45639af-531e-4b39-b369-560cda37b718" (UID: "a45639af-531e-4b39-b369-560cda37b718"). InnerVolumeSpecName "tempo-data-science-tempostack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:51.434571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.434538 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q" pod "a45639af-531e-4b39-b369-560cda37b718" (UID: "a45639af-531e-4b39-b369-560cda37b718"). InnerVolumeSpecName "kube-api-access-w6b2q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:51.434673 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.434639 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key" pod "a45639af-531e-4b39-b369-560cda37b718" (UID: "a45639af-531e-4b39-b369-560cda37b718"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:51.434753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.434731 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm" pod "e05e424c-453f-4e24-9957-4bf3e3f79799" (UID: "e05e424c-453f-4e24-9957-4bf3e3f79799"). InnerVolumeSpecName "kube-api-access-ctmxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:51.435101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.435075 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls" pod "e05e424c-453f-4e24-9957-4bf3e3f79799" (UID: "e05e424c-453f-4e24-9957-4bf3e3f79799"). InnerVolumeSpecName "tempo-data-science-tempostack-querier-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:51.435146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.435087 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key" pod "e05e424c-453f-4e24-9957-4bf3e3f79799" (UID: "e05e424c-453f-4e24-9957-4bf3e3f79799"). InnerVolumeSpecName "storage-gcs-key". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:51.435300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.435282 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls" pod "a45639af-531e-4b39-b369-560cda37b718" (UID: "a45639af-531e-4b39-b369-560cda37b718"). InnerVolumeSpecName "tempo-data-science-tempostack-query-frontend-mtls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:51.533349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533324 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533348 6966 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-storage-gcs-key\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533358 6966 reconciler_common.go:299] "Volume detached for volume \"storage-gcs-key\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-storage-gcs-key\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533366 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6b2q\" (UniqueName: \"kubernetes.io/projected/a45639af-531e-4b39-b369-560cda37b718-kube-api-access-w6b2q\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533375 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-query-frontend-mtls\" (UniqueName: \"kubernetes.io/secret/a45639af-531e-4b39-b369-560cda37b718-tempo-data-science-tempostack-query-frontend-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533384 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/a45639af-531e-4b39-b369-560cda37b718-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533393 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-data-science-tempostack-querier-mtls\" (UniqueName: \"kubernetes.io/secret/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-data-science-tempostack-querier-mtls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533401 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctmxm\" (UniqueName: \"kubernetes.io/projected/e05e424c-453f-4e24-9957-4bf3e3f79799-kube-api-access-ctmxm\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.533500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.533410 6966 reconciler_common.go:299] "Volume detached for volume \"tempo-conf\" (UniqueName: \"kubernetes.io/configmap/e05e424c-453f-4e24-9957-4bf3e3f79799-tempo-conf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:44:51.592519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.592485 6966 generic.go:356] "Generic (PLEG): container finished" podID="a45639af-531e-4b39-b369-560cda37b718" containerID="b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb" exitCode=137 Oct 30 18:44:51.592632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.592533 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" Oct 30 18:44:51.592632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.592558 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" event={"ID":"a45639af-531e-4b39-b369-560cda37b718","Type":"ContainerDied","Data":"b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb"} Oct 30 18:44:51.592632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.592611 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg" event={"ID":"a45639af-531e-4b39-b369-560cda37b718","Type":"ContainerDied","Data":"a76dfa0550cb2540da3965e568eb48cd7badbcd5fa0d0d2129cafd8fab0e87d5"} Oct 30 18:44:51.592806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.592633 6966 scope.go:122] "RemoveContainer" containerID="b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb" Oct 30 18:44:51.594257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.594142 6966 generic.go:356] "Generic (PLEG): container finished" podID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerID="d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d" exitCode=137 Oct 30 18:44:51.594257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.594184 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" event={"ID":"e05e424c-453f-4e24-9957-4bf3e3f79799","Type":"ContainerDied","Data":"d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d"} Oct 30 18:44:51.594257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.594195 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" Oct 30 18:44:51.594257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.594225 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7" event={"ID":"e05e424c-453f-4e24-9957-4bf3e3f79799","Type":"ContainerDied","Data":"5c2e5f825dc3c45ad4184595269f3d411901069a08b368aa79c92300054fba1a"} Oct 30 18:44:51.602960 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.602945 6966 scope.go:122] "RemoveContainer" containerID="b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb" Oct 30 18:44:51.603200 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:51.603181 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb\": container with ID starting with b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb not found: ID does not exist" containerID="b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb" Oct 30 18:44:51.603255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.603207 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb"} err="failed to get container status \"b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb\": rpc error: code = NotFound desc = could not find container \"b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb\": container with ID starting with b899d5f72500248f6e5761c2878a4715df24c428f33329094ca3ac4ae2fc59eb not found: ID does not exist" Oct 30 18:44:51.603255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.603224 6966 scope.go:122] "RemoveContainer" containerID="d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d" Oct 30 18:44:51.611838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.611822 6966 scope.go:122] "RemoveContainer" containerID="d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d" Oct 30 18:44:51.612082 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:51.612065 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d\": container with ID starting with d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d not found: ID does not exist" containerID="d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d" Oct 30 18:44:51.612142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.612093 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d"} err="failed to get container status \"d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d\": rpc error: code = NotFound desc = could not find container \"d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d\": container with ID starting with d437e810d74bb39190bb896738765aea0c160a2155d760a4c789977789a2c65d not found: ID does not exist" Oct 30 18:44:51.641657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.641634 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7"] Oct 30 18:44:51.652670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.652647 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-querier-7ddcfdcc88-mwbm7"] Oct 30 18:44:51.679791 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.679768 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg"] Oct 30 18:44:51.683770 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:51.683716 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/tempo-data-science-tempostack-query-frontend-6674686c47-q55lg"] Oct 30 18:44:52.248771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.248732 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:52.460937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.460846 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:52.474403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474376 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq"] Oct 30 18:44:52.474758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474745 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6ef81d08-ae85-4d6d-940a-9f2375ee38a9" containerName="korrel8r" Oct 30 18:44:52.474806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474760 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef81d08-ae85-4d6d-940a-9f2375ee38a9" containerName="korrel8r" Oct 30 18:44:52.474806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474771 6966 cpu_manager.go:414] "containerMap: removing container" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" Oct 30 18:44:52.474806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474779 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" Oct 30 18:44:52.474806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474790 6966 cpu_manager.go:414] "containerMap: removing container" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" Oct 30 18:44:52.474806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474795 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" Oct 30 18:44:52.474955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474858 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef81d08-ae85-4d6d-940a-9f2375ee38a9" containerName="korrel8r" Oct 30 18:44:52.474955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474867 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="a45639af-531e-4b39-b369-560cda37b718" containerName="tempo" Oct 30 18:44:52.474955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.474873 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" containerName="tempo" Oct 30 18:44:52.479805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.479787 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.500797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.500777 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq"] Oct 30 18:44:52.520390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.520370 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x"] Oct 30 18:44:52.524411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.524391 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.546995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.546976 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x"] Oct 30 18:44:52.563053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.563031 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx"] Oct 30 18:44:52.566524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.566503 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.586386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586358 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:44:52.586386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586356 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:44:52.586570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586388 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:44:52.586570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586418 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:44:52.586570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586427 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:44:52.586570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586390 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:44:52.586570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.586498 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:44:52.588287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.588264 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:44:52.588426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.588406 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:44:52.588526 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.588483 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:44:52.588526 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.588505 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-v7td7\"" Oct 30 18:44:52.588637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.588569 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-qgsl9\"" Oct 30 18:44:52.588637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.588578 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-lx22f\"" Oct 30 18:44:52.589783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.589765 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx"] Oct 30 18:44:52.639162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639134 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.639162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639163 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.639333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639190 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.639333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639206 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.639333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639263 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjk6h\" (UniqueName: \"kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.639333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639287 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.639333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639315 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.639644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639485 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.639644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639529 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.639644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639557 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.639785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.639665 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.666960 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.666938 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:52.740189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740138 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.740189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740165 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjk6h\" (UniqueName: \"kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.740365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740187 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.740365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740210 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.740365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740259 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.740365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740303 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.740365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740329 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.740665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740391 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.740665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740442 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.740665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740494 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.740665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740542 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.740665 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:52.740578 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 18:44:52.740665 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:44:52.740653 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls podName:7eea3adb-ea72-4956-931c-631340a39c7a nodeName:}" failed. No retries permitted until 2026-10-30 18:44:53.240634635 +0000 UTC m=+2855.946819969 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls") pod "korrel8r-5cbcc89b8f-qvjqx" (UID: "7eea3adb-ea72-4956-931c-631340a39c7a") : secret "korrel8r-tls" not found Oct 30 18:44:52.740996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.740949 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.741088 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.741066 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.741155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.741128 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.741425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.741400 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.741573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.741553 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.742905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.742887 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.742984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.742910 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.762891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.762869 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-82fjq\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.766014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.765988 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:52.766247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.766230 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjk6h\" (UniqueName: \"kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h\") pod \"data-science-thanos-querier-proxy-6998ff8b98-6rk5x\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.797108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.797089 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:44:52.841548 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.841521 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:44:52.872097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:52.872029 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.097116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.096996 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq"] Oct 30 18:44:53.101220 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:44:53.101193 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46a0366_f077_4f5e_a752_d8e6def549e4.slice/crio-3660259f7ece35d9fe4f88a69c51feb6a021d4111cb073e347cf98407118eb5e WatchSource:0}: Error finding container 3660259f7ece35d9fe4f88a69c51feb6a021d4111cb073e347cf98407118eb5e: Status 404 returned error can't find the container with id 3660259f7ece35d9fe4f88a69c51feb6a021d4111cb073e347cf98407118eb5e Oct 30 18:44:53.127876 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.127850 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.150027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.150004 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x"] Oct 30 18:44:53.154447 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:44:53.154425 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8318a332_6b14_4a3b_ae4c_8d1778e1ad31.slice/crio-80b0fed09a6f83fb8b83fde37ec665476aa5e769fcb3c406e26bfaeedc6811ee WatchSource:0}: Error finding container 80b0fed09a6f83fb8b83fde37ec665476aa5e769fcb3c406e26bfaeedc6811ee: Status 404 returned error can't find the container with id 80b0fed09a6f83fb8b83fde37ec665476aa5e769fcb3c406e26bfaeedc6811ee Oct 30 18:44:53.245385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.245354 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:53.247807 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.247786 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\") pod \"korrel8r-5cbcc89b8f-qvjqx\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:53.399549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.399521 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.411259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.411237 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a45639af-531e-4b39-b369-560cda37b718" path="/var/lib/kubelet/pods/a45639af-531e-4b39-b369-560cda37b718/volumes" Oct 30 18:44:53.411664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.411650 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e05e424c-453f-4e24-9957-4bf3e3f79799" path="/var/lib/kubelet/pods/e05e424c-453f-4e24-9957-4bf3e3f79799/volumes" Oct 30 18:44:53.478476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.478434 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:53.608532 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.608448 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:53.608856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.608828 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerStarted","Data":"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790"} Oct 30 18:44:53.608990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.608899 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerStarted","Data":"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd"} Oct 30 18:44:53.608990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.608936 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerStarted","Data":"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268"} Oct 30 18:44:53.608990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.608969 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerStarted","Data":"3660259f7ece35d9fe4f88a69c51feb6a021d4111cb073e347cf98407118eb5e"} Oct 30 18:44:53.610893 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.610872 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerStarted","Data":"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089"} Oct 30 18:44:53.610989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.610921 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerStarted","Data":"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d"} Oct 30 18:44:53.610989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.610948 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerStarted","Data":"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc"} Oct 30 18:44:53.610989 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.610971 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerStarted","Data":"80b0fed09a6f83fb8b83fde37ec665476aa5e769fcb3c406e26bfaeedc6811ee"} Oct 30 18:44:53.649959 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.649912 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" podStartSLOduration=1.6498972410000001 podStartE2EDuration="1.649897241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:53.649177264 +0000 UTC m=+2856.355362621" watchObservedRunningTime="2026-10-30 18:44:53.649897241 +0000 UTC m=+2856.356082600" Oct 30 18:44:53.690868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.690810 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" podStartSLOduration=1.690792852 podStartE2EDuration="1.690792852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:53.688873707 +0000 UTC m=+2856.395059064" watchObservedRunningTime="2026-10-30 18:44:53.690792852 +0000 UTC m=+2856.396978210" Oct 30 18:44:53.777003 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.776973 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx"] Oct 30 18:44:53.782659 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:44:53.782635 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eea3adb_ea72_4956_931c_631340a39c7a.slice/crio-a1a89cf8a3d26babb1fa368eb6e26f12f3f9be252dd56b236aab200415c513e0 WatchSource:0}: Error finding container a1a89cf8a3d26babb1fa368eb6e26f12f3f9be252dd56b236aab200415c513e0: Status 404 returned error can't find the container with id a1a89cf8a3d26babb1fa368eb6e26f12f3f9be252dd56b236aab200415c513e0 Oct 30 18:44:53.926328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.926244 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:44:53.930713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.930688 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:53.953627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.953599 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:44:53.954985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.954964 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:44:53.965242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.965210 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:44:53.965242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.965234 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:44:53.965394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.965291 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:44:53.965394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.965348 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:44:53.965485 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:53.965411 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:44:54.005040 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.005011 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:54.052491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.052444 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.052616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.052552 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.052660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.052615 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.052726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.052675 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.052726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.052698 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.052726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.052721 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59w68\" (UniqueName: \"kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.153552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.153521 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.153695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.153557 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59w68\" (UniqueName: \"kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.153695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.153595 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.153695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.153621 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.153695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.153651 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.153936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.153803 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.154580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.154553 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.156138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.156113 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.156250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.156195 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.156319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.156296 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.156360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.156304 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.176490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.176426 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59w68\" (UniqueName: \"kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68\") pod \"data-science-collector-collector-1\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.217509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.217485 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:44:54.245859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.245840 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:44:54.550103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.550029 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:44:54.555388 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:44:54.555363 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cc1e9df_ff4e_459a_9860_b23f08d00544.slice/crio-5bd10b967a39b91319ed0dcc3493b828469e8b173b4b7259dd0101b870c5148e WatchSource:0}: Error finding container 5bd10b967a39b91319ed0dcc3493b828469e8b173b4b7259dd0101b870c5148e: Status 404 returned error can't find the container with id 5bd10b967a39b91319ed0dcc3493b828469e8b173b4b7259dd0101b870c5148e Oct 30 18:44:54.557507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.557478 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:44:54.557588 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.557545 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:44:54.620022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.619995 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" event={"ID":"7eea3adb-ea72-4956-931c-631340a39c7a","Type":"ContainerStarted","Data":"ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d"} Oct 30 18:44:54.620146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.620049 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" event={"ID":"7eea3adb-ea72-4956-931c-631340a39c7a","Type":"ContainerStarted","Data":"a1a89cf8a3d26babb1fa368eb6e26f12f3f9be252dd56b236aab200415c513e0"} Oct 30 18:44:54.620146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.620099 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:44:54.621690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.621657 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8cc1e9df-ff4e-459a-9860-b23f08d00544","Type":"ContainerStarted","Data":"5bd10b967a39b91319ed0dcc3493b828469e8b173b4b7259dd0101b870c5148e"} Oct 30 18:44:54.660654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:54.660606 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" podStartSLOduration=2.660591853 podStartE2EDuration="2.660591853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:54.659136046 +0000 UTC m=+2857.365321405" watchObservedRunningTime="2026-10-30 18:44:54.660591853 +0000 UTC m=+2857.366777209" Oct 30 18:44:55.631232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:55.631185 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8cc1e9df-ff4e-459a-9860-b23f08d00544","Type":"ContainerStarted","Data":"fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815"} Oct 30 18:44:55.672567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:44:55.672523 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.672510373 podStartE2EDuration="2.672510373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:55.671917529 +0000 UTC m=+2858.378102887" watchObservedRunningTime="2026-10-30 18:44:55.672510373 +0000 UTC m=+2858.378695730" Oct 30 18:45:01.014910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.014868 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:01.244795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.244762 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:01.493374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.493338 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:01.597079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.597039 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:45:01.602512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.602483 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.613492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.609863 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:45:01.614267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.614245 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.628629 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.628601 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:45:01.641517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.641492 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:45:01.686812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686769 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686838 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686874 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686887 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686910 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686924 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686930 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686936 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-gjhkk\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686946 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686936 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686968 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:45:01.687022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.686972 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-q4sgp\"" Oct 30 18:45:01.694942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.694901 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:45:01.695186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.695029 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:45:01.695186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.695097 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:45:01.715627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715594 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.715736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715639 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.715736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715666 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.715736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715719 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.715866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715781 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.715866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715831 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.715975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715907 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7vrw\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.715975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715964 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.716075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.715994 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.716075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716031 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.716154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716104 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.716154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716127 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.716228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716154 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7975x\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.716228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716176 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.716228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716220 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.716328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716245 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.716328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716275 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.716328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716301 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.716491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.716327 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.720817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.720793 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:01.817650 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817622 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.817836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817669 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.817836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817702 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.817836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817721 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817831 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817868 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817928 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817962 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7vrw\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.817998 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818020 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818058 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818111 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818124 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818137 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818195 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7975x\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818234 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818272 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818309 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818339 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.818647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818367 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.818854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.818705 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.819585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.819558 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.820271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.820024 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.820958 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.820933 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.823149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.822549 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.823149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.822737 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.823149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.822864 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.823149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.822999 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.823149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.823078 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.823149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.823110 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.823622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.823274 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.823622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.823505 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.823622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.823539 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.823932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.823912 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.824124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.824102 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.824379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.824356 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.840499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.840452 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7975x\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:01.841311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.841292 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7vrw\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:01.924640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:01.924603 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:02.240249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:02.240138 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:45:02.244712 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:45:02.244687 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dfc0fe_85a3_43d1_a38b_4db33b4fb760.slice/crio-e8aa9e72ad3599ff707424742d191ef3be70fda6fce700a7b92d09c3a0414050 WatchSource:0}: Error finding container e8aa9e72ad3599ff707424742d191ef3be70fda6fce700a7b92d09c3a0414050: Status 404 returned error can't find the container with id e8aa9e72ad3599ff707424742d191ef3be70fda6fce700a7b92d09c3a0414050 Oct 30 18:45:02.670819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:02.670778 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerStarted","Data":"e8aa9e72ad3599ff707424742d191ef3be70fda6fce700a7b92d09c3a0414050"} Oct 30 18:45:03.677757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:03.677721 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerStarted","Data":"44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9"} Oct 30 18:45:05.633874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:05.633842 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:45:05.854926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:05.854888 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:05.882635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:05.882604 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:05.955731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:05.955642 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:05.955902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:05.955779 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:05.969502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:05.969472 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") DevicePath \"csi-83df9dfad9fd2ac35dca1441af096f35871cdbdee4927aba3c5e3091d60e99b5\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:06.069646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.069610 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/26c702426f1aeab2e9e6116e994cbd362b1230f690bb6d32002e36ee94fa3aec/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:06.096480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.096439 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"prometheus-data-science-monitoringstack-1\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:06.136297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.136220 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:06.469302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.469154 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:45:06.474040 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:45:06.474010 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2c78515_c331_4bbb_8393_12c31fe0f297.slice/crio-04b8fe722d37cac546592e9a530028e3057b4d1e2f6459cf01f25376e3ec2c21 WatchSource:0}: Error finding container 04b8fe722d37cac546592e9a530028e3057b4d1e2f6459cf01f25376e3ec2c21: Status 404 returned error can't find the container with id 04b8fe722d37cac546592e9a530028e3057b4d1e2f6459cf01f25376e3ec2c21 Oct 30 18:45:06.696425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.696390 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerStarted","Data":"04b8fe722d37cac546592e9a530028e3057b4d1e2f6459cf01f25376e3ec2c21"} Oct 30 18:45:06.697984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.697958 6966 generic.go:356] "Generic (PLEG): container finished" podID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerID="44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9" exitCode=0 Oct 30 18:45:06.698091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:06.697991 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerDied","Data":"44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9"} Oct 30 18:45:07.704888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:07.704794 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerStarted","Data":"494022b41a57ea991283d3e76a2985cdd76dc34b4e4fa16e61a5225840e46f8a"} Oct 30 18:45:07.706846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:07.706819 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerStarted","Data":"b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397"} Oct 30 18:45:07.706966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:07.706872 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerStarted","Data":"0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64"} Oct 30 18:45:07.707070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:07.707054 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:07.813031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:07.812986 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.42829778 podStartE2EDuration="6.812974704s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.384676924s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:07.811578669 +0000 UTC m=+2870.517764025" watchObservedRunningTime="2026-10-30 18:45:07.812974704 +0000 UTC m=+2870.519160060" Oct 30 18:45:10.725197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:10.725105 6966 generic.go:356] "Generic (PLEG): container finished" podID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerID="494022b41a57ea991283d3e76a2985cdd76dc34b4e4fa16e61a5225840e46f8a" exitCode=0 Oct 30 18:45:10.725593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:10.725183 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerDied","Data":"494022b41a57ea991283d3e76a2985cdd76dc34b4e4fa16e61a5225840e46f8a"} Oct 30 18:45:11.734329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:11.734292 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerStarted","Data":"931f6a54289e74fe2c693edf95ad86bcecaf078a6295c8691e189481d4345e33"} Oct 30 18:45:11.734802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:11.734366 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerStarted","Data":"f96c7004aeb776c9bee90330eeab33abe7989342f2303dbfa70f1c3771bdd418"} Oct 30 18:45:11.734802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:11.734394 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerStarted","Data":"3a806099266f27e5953ce766ca29e851189da29453ca9cca49238034b21b1d91"} Oct 30 18:45:11.817436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:11.817383 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podStartSLOduration=7.333401783 podStartE2EDuration="10.817372163s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.48397038s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:11.806061823 +0000 UTC m=+2874.512247177" watchObservedRunningTime="2026-10-30 18:45:11.817372163 +0000 UTC m=+2874.523557519" Oct 30 18:45:13.713299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:13.713267 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:45:16.137181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:16.137147 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:17.202174 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:45:17.202135 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:45:21.136435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:21.136399 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:21.141806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:21.141785 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:21.797790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:21.797763 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:45:22.397504 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:22.397453 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:23.405959 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:23.405921 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:35.922975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:35.922895 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:36.490581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:36.490550 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:36.719059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:36.719031 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:37.980280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:37.980224 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:45:37.980764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:37.980508 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8cc1e9df-ff4e-459a-9860-b23f08d00544" containerName="otc-container" containerID="cri-o://fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815" gracePeriod=30 Oct 30 18:45:38.222966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:38.222937 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:38.446620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:38.446587 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:45:40.330544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.330518 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:45:40.338470 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.338435 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68\" (UniqueName: \"kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68\") pod \"8cc1e9df-ff4e-459a-9860-b23f08d00544\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " Oct 30 18:45:40.338576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.338503 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert\") pod \"8cc1e9df-ff4e-459a-9860-b23f08d00544\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " Oct 30 18:45:40.338576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.338538 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs\") pod \"8cc1e9df-ff4e-459a-9860-b23f08d00544\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " Oct 30 18:45:40.338696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.338583 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs\") pod \"8cc1e9df-ff4e-459a-9860-b23f08d00544\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " Oct 30 18:45:40.338696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.338636 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal\") pod \"8cc1e9df-ff4e-459a-9860-b23f08d00544\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " Oct 30 18:45:40.338696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.338674 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs\") pod \"8cc1e9df-ff4e-459a-9860-b23f08d00544\" (UID: \"8cc1e9df-ff4e-459a-9860-b23f08d00544\") " Oct 30 18:45:40.339026 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.339000 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal" pod "8cc1e9df-ff4e-459a-9860-b23f08d00544" (UID: "8cc1e9df-ff4e-459a-9860-b23f08d00544"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:40.340748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.340720 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert" pod "8cc1e9df-ff4e-459a-9860-b23f08d00544" (UID: "8cc1e9df-ff4e-459a-9860-b23f08d00544"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:40.340871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.340797 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68" pod "8cc1e9df-ff4e-459a-9860-b23f08d00544" (UID: "8cc1e9df-ff4e-459a-9860-b23f08d00544"). InnerVolumeSpecName "kube-api-access-59w68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:40.341042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.341023 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs" pod "8cc1e9df-ff4e-459a-9860-b23f08d00544" (UID: "8cc1e9df-ff4e-459a-9860-b23f08d00544"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:40.341131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.341100 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs" pod "8cc1e9df-ff4e-459a-9860-b23f08d00544" (UID: "8cc1e9df-ff4e-459a-9860-b23f08d00544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:40.341196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.341172 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs" pod "8cc1e9df-ff4e-459a-9860-b23f08d00544" (UID: "8cc1e9df-ff4e-459a-9860-b23f08d00544"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:40.439604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.439570 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59w68\" (UniqueName: \"kubernetes.io/projected/8cc1e9df-ff4e-459a-9860-b23f08d00544-kube-api-access-59w68\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.439604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.439604 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.439824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.439622 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.439824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.439635 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.439824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.439648 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8cc1e9df-ff4e-459a-9860-b23f08d00544-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.439824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.439662 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8cc1e9df-ff4e-459a-9860-b23f08d00544-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:45:40.901645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.901614 6966 generic.go:356] "Generic (PLEG): container finished" podID="8cc1e9df-ff4e-459a-9860-b23f08d00544" containerID="fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815" exitCode=0 Oct 30 18:45:40.901797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.901657 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:45:40.901797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.901695 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8cc1e9df-ff4e-459a-9860-b23f08d00544","Type":"ContainerDied","Data":"fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815"} Oct 30 18:45:40.901797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.901748 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8cc1e9df-ff4e-459a-9860-b23f08d00544","Type":"ContainerDied","Data":"5bd10b967a39b91319ed0dcc3493b828469e8b173b4b7259dd0101b870c5148e"} Oct 30 18:45:40.901797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.901771 6966 scope.go:122] "RemoveContainer" containerID="fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815" Oct 30 18:45:40.911293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.911276 6966 scope.go:122] "RemoveContainer" containerID="fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815" Oct 30 18:45:40.911570 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:45:40.911550 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815\": container with ID starting with fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815 not found: ID does not exist" containerID="fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815" Oct 30 18:45:40.911631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.911578 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815"} err="failed to get container status \"fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815\": rpc error: code = NotFound desc = could not find container \"fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815\": container with ID starting with fa2d8ea4930cfc8888baeb201381e5dc938791c2b6e794d39dca78d0af0c3815 not found: ID does not exist" Oct 30 18:45:40.952456 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.952431 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:45:40.963512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:40.963491 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:45:41.411803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:41.411773 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc1e9df-ff4e-459a-9860-b23f08d00544" path="/var/lib/kubelet/pods/8cc1e9df-ff4e-459a-9860-b23f08d00544/volumes" Oct 30 18:45:47.200647 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:45:47.200608 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:45:59.893268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:45:59.893215 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:00.148086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:00.148004 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:09.058209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:09.058165 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:15.033336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:15.033299 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:15.264254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:15.264212 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:17.199358 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:46:17.199321 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:46:22.169564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.169524 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:22.445179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.445094 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:22.446935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.446915 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf"] Oct 30 18:46:22.447490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.447471 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8cc1e9df-ff4e-459a-9860-b23f08d00544" containerName="otc-container" Oct 30 18:46:22.447490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.447489 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc1e9df-ff4e-459a-9860-b23f08d00544" containerName="otc-container" Oct 30 18:46:22.447657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.447610 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc1e9df-ff4e-459a-9860-b23f08d00544" containerName="otc-container" Oct 30 18:46:22.450632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.450614 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.471353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.471284 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf"] Oct 30 18:46:22.592885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.592852 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.593046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.592994 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.593046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.593023 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmjwc\" (UniqueName: \"kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.674181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.674145 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:22.693997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.693968 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.694126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.694000 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmjwc\" (UniqueName: \"kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.694185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.694129 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.694912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.694893 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.696621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.696569 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.717218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.717198 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmjwc\" (UniqueName: \"kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc\") pod \"korrel8r-66d6f7f696-vghzf\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:22.763352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:22.763330 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:23.058850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:23.058756 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf"] Oct 30 18:46:23.063389 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:46:23.063365 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29389b55_6614_4f25_b7da_178048d00179.slice/crio-dd432cc08c87171afbe962b6c1001e48909b72347adbb149642cde594d114adf WatchSource:0}: Error finding container dd432cc08c87171afbe962b6c1001e48909b72347adbb149642cde594d114adf: Status 404 returned error can't find the container with id dd432cc08c87171afbe962b6c1001e48909b72347adbb149642cde594d114adf Oct 30 18:46:23.122444 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:23.122419 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" event={"ID":"29389b55-6614-4f25-b7da-178048d00179","Type":"ContainerStarted","Data":"dd432cc08c87171afbe962b6c1001e48909b72347adbb149642cde594d114adf"} Oct 30 18:46:23.540824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:23.540790 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:24.129404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:24.129370 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" event={"ID":"29389b55-6614-4f25-b7da-178048d00179","Type":"ContainerStarted","Data":"6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be"} Oct 30 18:46:24.129621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:24.129508 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:24.173934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:24.173888 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" podStartSLOduration=2.173875629 podStartE2EDuration="2.173875629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:24.172852876 +0000 UTC m=+2946.879038233" watchObservedRunningTime="2026-10-30 18:46:24.173875629 +0000 UTC m=+2946.880060988" Oct 30 18:46:24.204529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:24.204501 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:24.450570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:24.450485 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:25.581813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:25.581776 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:25.809992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:25.809956 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:30.523647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.523613 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:30.527593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.527574 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.552942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.552922 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:46:30.553067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.553047 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:46:30.553116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.553095 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:46:30.553116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.553104 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:46:30.564171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.564148 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:46:30.564297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.564177 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:46:30.565972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.565950 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:30.667128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.667093 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtcfv\" (UniqueName: \"kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.667265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.667146 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.667310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.667274 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.667343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.667312 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.667492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.667451 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.667556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.667516 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.768078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.768050 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.768216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.768097 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.768216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.768185 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.768317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.768228 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.768317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.768259 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtcfv\" (UniqueName: \"kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.768317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.768299 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.769202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.769175 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.770772 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.770750 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.770873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.770795 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.771079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.771061 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.771188 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.771167 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.791097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.791035 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtcfv\" (UniqueName: \"kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv\") pod \"data-science-collector-collector-0\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:30.841254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:30.841227 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:31.140854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:31.140821 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:31.145208 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:46:31.145177 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod234c4c32_5936_4dc8_bb9f_15e8d0e0d0c3.slice/crio-8ea435258e142d1ca3200667ee337834a818c221e451704b2e76ed36756ed3fb WatchSource:0}: Error finding container 8ea435258e142d1ca3200667ee337834a818c221e451704b2e76ed36756ed3fb: Status 404 returned error can't find the container with id 8ea435258e142d1ca3200667ee337834a818c221e451704b2e76ed36756ed3fb Oct 30 18:46:31.147424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:31.147392 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:46:31.147523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:31.147453 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:46:31.172179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:31.172150 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3","Type":"ContainerStarted","Data":"8ea435258e142d1ca3200667ee337834a818c221e451704b2e76ed36756ed3fb"} Oct 30 18:46:32.179327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:32.179294 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3","Type":"ContainerStarted","Data":"c819e1265005f5d05be5efedc727c55c6bc02e529c30b7207ee9aa20fc449069"} Oct 30 18:46:32.221491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:32.221416 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.221399535 podStartE2EDuration="2.221399535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:32.219551826 +0000 UTC m=+2954.925737219" watchObservedRunningTime="2026-10-30 18:46:32.221399535 +0000 UTC m=+2954.927584894" Oct 30 18:46:35.139568 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.139540 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:35.210515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.210484 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx"] Oct 30 18:46:35.211133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.211102 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" podUID="7eea3adb-ea72-4956-931c-631340a39c7a" containerName="korrel8r" containerID="cri-o://ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d" gracePeriod=30 Oct 30 18:46:35.465562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.465539 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:46:35.504898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.504871 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\" (UniqueName: \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\") pod \"7eea3adb-ea72-4956-931c-631340a39c7a\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " Oct 30 18:46:35.505045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.504945 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2\") pod \"7eea3adb-ea72-4956-931c-631340a39c7a\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " Oct 30 18:46:35.505045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.505024 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config\" (UniqueName: \"kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config\") pod \"7eea3adb-ea72-4956-931c-631340a39c7a\" (UID: \"7eea3adb-ea72-4956-931c-631340a39c7a\") " Oct 30 18:46:35.505591 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.505547 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config" pod "7eea3adb-ea72-4956-931c-631340a39c7a" (UID: "7eea3adb-ea72-4956-931c-631340a39c7a"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:35.505970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.505947 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7eea3adb-ea72-4956-931c-631340a39c7a-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:35.507413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.507386 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls" pod "7eea3adb-ea72-4956-931c-631340a39c7a" (UID: "7eea3adb-ea72-4956-931c-631340a39c7a"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:35.507692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.507661 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2" pod "7eea3adb-ea72-4956-931c-631340a39c7a" (UID: "7eea3adb-ea72-4956-931c-631340a39c7a"). InnerVolumeSpecName "kube-api-access-rlln2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:35.606577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.606546 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/7eea3adb-ea72-4956-931c-631340a39c7a-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:35.606577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:35.606574 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlln2\" (UniqueName: \"kubernetes.io/projected/7eea3adb-ea72-4956-931c-631340a39c7a-kube-api-access-rlln2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:36.200988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.200954 6966 generic.go:356] "Generic (PLEG): container finished" podID="7eea3adb-ea72-4956-931c-631340a39c7a" containerID="ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d" exitCode=0 Oct 30 18:46:36.201377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.201004 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" Oct 30 18:46:36.201377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.201038 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" event={"ID":"7eea3adb-ea72-4956-931c-631340a39c7a","Type":"ContainerDied","Data":"ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d"} Oct 30 18:46:36.201377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.201088 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx" event={"ID":"7eea3adb-ea72-4956-931c-631340a39c7a","Type":"ContainerDied","Data":"a1a89cf8a3d26babb1fa368eb6e26f12f3f9be252dd56b236aab200415c513e0"} Oct 30 18:46:36.201377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.201109 6966 scope.go:122] "RemoveContainer" containerID="ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d" Oct 30 18:46:36.210509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.210486 6966 scope.go:122] "RemoveContainer" containerID="ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d" Oct 30 18:46:36.210779 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:46:36.210760 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d\": container with ID starting with ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d not found: ID does not exist" containerID="ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d" Oct 30 18:46:36.210839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.210788 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d"} err="failed to get container status \"ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d\": rpc error: code = NotFound desc = could not find container \"ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d\": container with ID starting with ca7a3ed2eeb7fc6d4aedf299762941449bb056d624aa3eef5ead12e71e736d3d not found: ID does not exist" Oct 30 18:46:36.250278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.250248 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx"] Oct 30 18:46:36.260930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:36.260910 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-qvjqx"] Oct 30 18:46:37.411854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:37.411824 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eea3adb-ea72-4956-931c-631340a39c7a" path="/var/lib/kubelet/pods/7eea3adb-ea72-4956-931c-631340a39c7a/volumes" Oct 30 18:46:39.547095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.547054 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:39.834419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.834386 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:39.836577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.836555 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9"] Oct 30 18:46:39.837090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.837076 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7eea3adb-ea72-4956-931c-631340a39c7a" containerName="korrel8r" Oct 30 18:46:39.837140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.837092 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eea3adb-ea72-4956-931c-631340a39c7a" containerName="korrel8r" Oct 30 18:46:39.837189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.837177 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eea3adb-ea72-4956-931c-631340a39c7a" containerName="korrel8r" Oct 30 18:46:39.841686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.841667 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:39.861474 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.861435 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9"] Oct 30 18:46:39.942405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.942374 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmspd\" (UniqueName: \"kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:39.942582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.942503 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:39.942582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:39.942547 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.043405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.043373 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.043573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.043415 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.043573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.043517 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmspd\" (UniqueName: \"kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.044265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.044244 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.045767 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.045746 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.065709 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.065682 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmspd\" (UniqueName: \"kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd\") pod \"korrel8r-5cbcc89b8f-2dbl9\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.154836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.154781 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:40.454229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:40.454147 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9"] Oct 30 18:46:41.234858 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:41.234825 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" event={"ID":"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881","Type":"ContainerStarted","Data":"7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7"} Oct 30 18:46:41.235254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:41.234876 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" event={"ID":"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881","Type":"ContainerStarted","Data":"e41db4f825e73add930ab55d5b0bf86381af93fb51992da639d5cdc82015eb14"} Oct 30 18:46:41.235254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:41.234908 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:41.288827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:41.288782 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" podStartSLOduration=2.28876907 podStartE2EDuration="2.28876907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:41.28023646 +0000 UTC m=+2963.986421818" watchObservedRunningTime="2026-10-30 18:46:41.28876907 +0000 UTC m=+2963.994954471" Oct 30 18:46:41.336515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:41.336479 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:41.638014 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:41.637984 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:42.270725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:42.270695 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:44.028491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.028426 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:44.028950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.028681 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" containerName="otc-container" containerID="cri-o://c819e1265005f5d05be5efedc727c55c6bc02e529c30b7207ee9aa20fc449069" gracePeriod=30 Oct 30 18:46:44.252951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.252919 6966 generic.go:356] "Generic (PLEG): container finished" podID="234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" containerID="c819e1265005f5d05be5efedc727c55c6bc02e529c30b7207ee9aa20fc449069" exitCode=0 Oct 30 18:46:44.253091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.252994 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3","Type":"ContainerDied","Data":"c819e1265005f5d05be5efedc727c55c6bc02e529c30b7207ee9aa20fc449069"} Oct 30 18:46:44.280187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.280138 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:44.381632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.381602 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal\") pod \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " Oct 30 18:46:44.381792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.381685 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv\" (UniqueName: \"kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv\") pod \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " Oct 30 18:46:44.381792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.381776 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs\") pod \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " Oct 30 18:46:44.381880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.381819 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert\") pod \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " Oct 30 18:46:44.381932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.381888 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs\") pod \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " Oct 30 18:46:44.381932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.381913 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs\") pod \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\" (UID: \"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3\") " Oct 30 18:46:44.382046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.382023 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal" pod "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" (UID: "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:44.382260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.382235 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:44.383907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.383878 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs" pod "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" (UID: "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:44.383907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.383896 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv" pod "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" (UID: "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3"). InnerVolumeSpecName "kube-api-access-rtcfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:44.384108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.384007 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs" pod "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" (UID: "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:44.384309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.384290 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert" pod "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" (UID: "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:44.384411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.384312 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs" pod "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" (UID: "234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:44.483498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.483235 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:44.483498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.483259 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:44.483498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.483275 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtcfv\" (UniqueName: \"kubernetes.io/projected/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-kube-api-access-rtcfv\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:44.483498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.483284 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:44.483498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:44.483294 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:45.259969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:45.259935 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3","Type":"ContainerDied","Data":"8ea435258e142d1ca3200667ee337834a818c221e451704b2e76ed36756ed3fb"} Oct 30 18:46:45.259969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:45.259962 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:46:45.260479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:45.259999 6966 scope.go:122] "RemoveContainer" containerID="c819e1265005f5d05be5efedc727c55c6bc02e529c30b7207ee9aa20fc449069" Oct 30 18:46:45.314931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:45.314906 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:45.318917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:45.318894 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:46:45.411771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:45.411741 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" path="/var/lib/kubelet/pods/234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3/volumes" Oct 30 18:46:47.199543 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:46:47.199510 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:46:50.458753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:50.458718 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:52.242778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.242747 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:46:52.311811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.311782 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf"] Oct 30 18:46:52.312266 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.312208 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" podUID="29389b55-6614-4f25-b7da-178048d00179" containerName="korrel8r" containerID="cri-o://6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be" gracePeriod=30 Oct 30 18:46:52.557020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.556997 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:52.653435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.653403 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc\" (UniqueName: \"kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc\") pod \"29389b55-6614-4f25-b7da-178048d00179\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " Oct 30 18:46:52.653613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.653448 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config\" (UniqueName: \"kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config\") pod \"29389b55-6614-4f25-b7da-178048d00179\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " Oct 30 18:46:52.653613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.653507 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls\" (UniqueName: \"kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls\") pod \"29389b55-6614-4f25-b7da-178048d00179\" (UID: \"29389b55-6614-4f25-b7da-178048d00179\") " Oct 30 18:46:52.653925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.653900 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config" pod "29389b55-6614-4f25-b7da-178048d00179" (UID: "29389b55-6614-4f25-b7da-178048d00179"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:52.655615 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.655587 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls" pod "29389b55-6614-4f25-b7da-178048d00179" (UID: "29389b55-6614-4f25-b7da-178048d00179"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:52.655727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.655701 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc" pod "29389b55-6614-4f25-b7da-178048d00179" (UID: "29389b55-6614-4f25-b7da-178048d00179"). InnerVolumeSpecName "kube-api-access-nmjwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:52.754562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.754532 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmjwc\" (UniqueName: \"kubernetes.io/projected/29389b55-6614-4f25-b7da-178048d00179-kube-api-access-nmjwc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:52.754562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.754559 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/29389b55-6614-4f25-b7da-178048d00179-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:52.754562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:52.754568 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/29389b55-6614-4f25-b7da-178048d00179-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:46:53.303435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.303399 6966 generic.go:356] "Generic (PLEG): container finished" podID="29389b55-6614-4f25-b7da-178048d00179" containerID="6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be" exitCode=0 Oct 30 18:46:53.303862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.303449 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" Oct 30 18:46:53.303862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.303493 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" event={"ID":"29389b55-6614-4f25-b7da-178048d00179","Type":"ContainerDied","Data":"6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be"} Oct 30 18:46:53.303862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.303553 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf" event={"ID":"29389b55-6614-4f25-b7da-178048d00179","Type":"ContainerDied","Data":"dd432cc08c87171afbe962b6c1001e48909b72347adbb149642cde594d114adf"} Oct 30 18:46:53.303862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.303575 6966 scope.go:122] "RemoveContainer" containerID="6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be" Oct 30 18:46:53.313302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.313096 6966 scope.go:122] "RemoveContainer" containerID="6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be" Oct 30 18:46:53.313378 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:46:53.313359 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be\": container with ID starting with 6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be not found: ID does not exist" containerID="6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be" Oct 30 18:46:53.313423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.313388 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be"} err="failed to get container status \"6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be\": rpc error: code = NotFound desc = could not find container \"6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be\": container with ID starting with 6e7c8f5d235bf467b074f1c74a7111baf7698136ab04e5723291da3c5943b4be not found: ID does not exist" Oct 30 18:46:53.351930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.351908 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf"] Oct 30 18:46:53.362796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.362776 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vghzf"] Oct 30 18:46:53.411051 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:53.411027 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29389b55-6614-4f25-b7da-178048d00179" path="/var/lib/kubelet/pods/29389b55-6614-4f25-b7da-178048d00179/volumes" Oct 30 18:46:56.725598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:56.725561 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:56.944248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:56.944199 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:57.008681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.008612 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn"] Oct 30 18:46:57.009034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.009021 6966 cpu_manager.go:414] "containerMap: removing container" podUID="234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" containerName="otc-container" Oct 30 18:46:57.009080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.009036 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" containerName="otc-container" Oct 30 18:46:57.009157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.009146 6966 cpu_manager.go:414] "containerMap: removing container" podUID="29389b55-6614-4f25-b7da-178048d00179" containerName="korrel8r" Oct 30 18:46:57.009215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.009159 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29389b55-6614-4f25-b7da-178048d00179" containerName="korrel8r" Oct 30 18:46:57.009257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.009249 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="29389b55-6614-4f25-b7da-178048d00179" containerName="korrel8r" Oct 30 18:46:57.009304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.009259 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="234c4c32-5936-4dc8-bb9f-15e8d0e0d0c3" containerName="otc-container" Oct 30 18:46:57.013975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.013957 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.036237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.036214 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn"] Oct 30 18:46:57.092980 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.092945 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb7mz\" (UniqueName: \"kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.093133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.092998 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.093133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.093019 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.159632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.159608 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:57.193882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.193856 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb7mz\" (UniqueName: \"kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.194010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.193896 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.194010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.193929 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.194796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.194779 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.196288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.196270 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.216143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.216121 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb7mz\" (UniqueName: \"kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz\") pod \"korrel8r-66d6f7f696-vkvqn\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.326681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.326649 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:57.623979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:57.623909 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn"] Oct 30 18:46:57.628421 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:46:57.628395 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9351828f_758b_463e_a396_87ef93ac3ac9.slice/crio-2324a10d7ab2b5eaef9f781acd318f4b27540014d7e1ed17ad02bc21b6924b60 WatchSource:0}: Error finding container 2324a10d7ab2b5eaef9f781acd318f4b27540014d7e1ed17ad02bc21b6924b60: Status 404 returned error can't find the container with id 2324a10d7ab2b5eaef9f781acd318f4b27540014d7e1ed17ad02bc21b6924b60 Oct 30 18:46:58.329721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:58.329688 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" event={"ID":"9351828f-758b-463e-a396-87ef93ac3ac9","Type":"ContainerStarted","Data":"89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120"} Oct 30 18:46:58.330176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:58.329747 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" event={"ID":"9351828f-758b-463e-a396-87ef93ac3ac9","Type":"ContainerStarted","Data":"2324a10d7ab2b5eaef9f781acd318f4b27540014d7e1ed17ad02bc21b6924b60"} Oct 30 18:46:58.330176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:58.329880 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:46:58.369562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:58.369505 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" podStartSLOduration=2.3694892960000002 podStartE2EDuration="2.369489296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:58.36807848 +0000 UTC m=+2981.074263837" watchObservedRunningTime="2026-10-30 18:46:58.369489296 +0000 UTC m=+2981.075674659" Oct 30 18:46:58.651145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:58.651062 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:58.918191 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:58.918080 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:59.692265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:59.692224 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:46:59.910697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:46:59.910661 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:01.683570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.683539 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:01.687514 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.687491 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.712570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.712536 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:47:01.712690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.712578 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:47:01.712690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.712638 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-hsglb\"" Oct 30 18:47:01.712690 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.712659 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:47:01.712825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.712703 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:47:01.723715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.723695 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-6d9d4583\"" Oct 30 18:47:01.725322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.725303 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:01.733566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.733540 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.733705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.733580 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.733705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.733670 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.733705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.733702 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjbgt\" (UniqueName: \"kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.733867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.733732 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.733867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.733839 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.834582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.834556 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.834708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.834604 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.834708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.834624 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.834708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.834655 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.834708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.834675 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjbgt\" (UniqueName: \"kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.834708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.834697 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.835509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.835484 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.837118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.837096 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.837548 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.837527 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.837626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.837606 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.837886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.837868 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:01.857584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:01.857560 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjbgt\" (UniqueName: \"kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt\") pod \"data-science-collector-collector-0\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:02.001913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:02.001844 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:02.299565 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:02.299481 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:02.304063 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:02.304036 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0554f121_8d0b_44b1_b9d2_e5c8d795ee71.slice/crio-33a52846b56b9b4bcb1a051bf18de8281fcbd4794cd3655c7e5aed908f8b0128 WatchSource:0}: Error finding container 33a52846b56b9b4bcb1a051bf18de8281fcbd4794cd3655c7e5aed908f8b0128: Status 404 returned error can't find the container with id 33a52846b56b9b4bcb1a051bf18de8281fcbd4794cd3655c7e5aed908f8b0128 Oct 30 18:47:02.306186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:02.306148 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:47:02.306284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:02.306212 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:47:02.352521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:02.352489 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0554f121-8d0b-44b1-b9d2-e5c8d795ee71","Type":"ContainerStarted","Data":"33a52846b56b9b4bcb1a051bf18de8281fcbd4794cd3655c7e5aed908f8b0128"} Oct 30 18:47:03.360992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:03.360957 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0554f121-8d0b-44b1-b9d2-e5c8d795ee71","Type":"ContainerStarted","Data":"e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426"} Oct 30 18:47:03.413439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:03.413386 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-0" podStartSLOduration=2.413374258 podStartE2EDuration="2.413374258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:03.401774591 +0000 UTC m=+2986.107959949" watchObservedRunningTime="2026-10-30 18:47:03.413374258 +0000 UTC m=+2986.119559642" Oct 30 18:47:09.337072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.337046 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:47:09.412633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.412601 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9"] Oct 30 18:47:09.412795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.412781 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" podUID="1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" containerName="korrel8r" containerID="cri-o://7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7" gracePeriod=30 Oct 30 18:47:09.665102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.665074 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:47:09.805027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.804999 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config\" (UniqueName: \"kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config\") pod \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " Oct 30 18:47:09.805195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.805063 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd\" (UniqueName: \"kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd\") pod \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " Oct 30 18:47:09.805195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.805106 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls\" (UniqueName: \"kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls\") pod \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\" (UID: \"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881\") " Oct 30 18:47:09.805386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.805362 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config" pod "1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" (UID: "1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:09.807115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.807084 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd" pod "1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" (UID: "1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881"). InnerVolumeSpecName "kube-api-access-nmspd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:09.807270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.807251 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls" pod "1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" (UID: "1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:09.906162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.906096 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:09.906162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.906124 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:09.906162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:09.906137 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmspd\" (UniqueName: \"kubernetes.io/projected/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881-kube-api-access-nmspd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:10.399411 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.399373 6966 generic.go:356] "Generic (PLEG): container finished" podID="1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" containerID="7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7" exitCode=0 Oct 30 18:47:10.399853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.399417 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" Oct 30 18:47:10.399853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.399468 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" event={"ID":"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881","Type":"ContainerDied","Data":"7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7"} Oct 30 18:47:10.399853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.399522 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9" event={"ID":"1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881","Type":"ContainerDied","Data":"e41db4f825e73add930ab55d5b0bf86381af93fb51992da639d5cdc82015eb14"} Oct 30 18:47:10.399853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.399546 6966 scope.go:122] "RemoveContainer" containerID="7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7" Oct 30 18:47:10.409164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.409148 6966 scope.go:122] "RemoveContainer" containerID="7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7" Oct 30 18:47:10.409413 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:10.409396 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7\": container with ID starting with 7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7 not found: ID does not exist" containerID="7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7" Oct 30 18:47:10.409476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.409421 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7"} err="failed to get container status \"7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7\": rpc error: code = NotFound desc = could not find container \"7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7\": container with ID starting with 7c35d7573818444f8071c038e6c29631eb5324bcf47df0243d48b15c1d45d7c7 not found: ID does not exist" Oct 30 18:47:10.448236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.448205 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9"] Oct 30 18:47:10.459061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:10.459035 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-2dbl9"] Oct 30 18:47:11.410642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:11.410609 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" path="/var/lib/kubelet/pods/1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881/volumes" Oct 30 18:47:14.078201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.078168 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.302202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.302169 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj"] Oct 30 18:47:14.302707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.302682 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" containerName="korrel8r" Oct 30 18:47:14.302707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.302705 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" containerName="korrel8r" Oct 30 18:47:14.302903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.302825 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9ccd0c-bd6b-4916-a391-7fbe4c8bb881" containerName="korrel8r" Oct 30 18:47:14.307490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.307456 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.310226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.310181 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:14.330663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.330597 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj"] Oct 30 18:47:14.449292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.449265 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.449456 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.449310 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.449456 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.449402 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7mw8\" (UniqueName: \"kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.550380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.550349 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.550572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.550404 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7mw8\" (UniqueName: \"kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.550572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.550448 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.551101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.551071 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.552931 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.552909 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.574187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.574160 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7mw8\" (UniqueName: \"kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8\") pod \"korrel8r-5cbcc89b8f-ddpzj\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.620015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.619958 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:14.917924 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:14.917837 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj"] Oct 30 18:47:14.922575 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:14.922542 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7f702b_3fe7_4105_8a6e_e5d2f0d55f72.slice/crio-19941fb739269a2b6f9a7a2be2c9bb44fb734db80b30a3ccfed7989b7d35e57e WatchSource:0}: Error finding container 19941fb739269a2b6f9a7a2be2c9bb44fb734db80b30a3ccfed7989b7d35e57e: Status 404 returned error can't find the container with id 19941fb739269a2b6f9a7a2be2c9bb44fb734db80b30a3ccfed7989b7d35e57e Oct 30 18:47:15.426229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:15.426192 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" event={"ID":"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72","Type":"ContainerStarted","Data":"04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b"} Oct 30 18:47:15.426691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:15.426248 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" event={"ID":"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72","Type":"ContainerStarted","Data":"19941fb739269a2b6f9a7a2be2c9bb44fb734db80b30a3ccfed7989b7d35e57e"} Oct 30 18:47:15.426691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:15.426278 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:15.465715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:15.465669 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" podStartSLOduration=1.4656568380000001 podStartE2EDuration="1.465656838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:15.464086075 +0000 UTC m=+2998.170271434" watchObservedRunningTime="2026-10-30 18:47:15.465656838 +0000 UTC m=+2998.171842195" Oct 30 18:47:15.801095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:15.800932 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:16.045688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:16.045652 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:16.742091 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:16.742055 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:17.201235 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:17.201196 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:47:18.187652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:18.187621 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:18.188056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:18.187880 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-0" podUID="0554f121-8d0b-44b1-b9d2-e5c8d795ee71" containerName="otc-container" containerID="cri-o://e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426" gracePeriod=30 Oct 30 18:47:20.438524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.438497 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:20.453711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.453679 6966 generic.go:356] "Generic (PLEG): container finished" podID="0554f121-8d0b-44b1-b9d2-e5c8d795ee71" containerID="e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426" exitCode=0 Oct 30 18:47:20.453838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.453720 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-0" Oct 30 18:47:20.453838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.453740 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0554f121-8d0b-44b1-b9d2-e5c8d795ee71","Type":"ContainerDied","Data":"e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426"} Oct 30 18:47:20.453838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.453792 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-0" event={"ID":"0554f121-8d0b-44b1-b9d2-e5c8d795ee71","Type":"ContainerDied","Data":"33a52846b56b9b4bcb1a051bf18de8281fcbd4794cd3655c7e5aed908f8b0128"} Oct 30 18:47:20.453838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.453823 6966 scope.go:122] "RemoveContainer" containerID="e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426" Oct 30 18:47:20.463107 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.463092 6966 scope.go:122] "RemoveContainer" containerID="e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426" Oct 30 18:47:20.463394 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:20.463368 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426\": container with ID starting with e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426 not found: ID does not exist" containerID="e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426" Oct 30 18:47:20.463500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.463400 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426"} err="failed to get container status \"e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426\": rpc error: code = NotFound desc = could not find container \"e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426\": container with ID starting with e577a1b65c9dc79621bf2a0c8c4e8120b29dba24c05e5bd908419e0899780426 not found: ID does not exist" Oct 30 18:47:20.606347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606313 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert\") pod \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " Oct 30 18:47:20.606550 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606355 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal\" (UniqueName: \"kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal\") pod \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " Oct 30 18:47:20.606550 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606435 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs\") pod \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " Oct 30 18:47:20.606550 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606496 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt\" (UniqueName: \"kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt\") pod \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " Oct 30 18:47:20.606749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606551 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs\") pod \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " Oct 30 18:47:20.606749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606567 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs\") pod \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\" (UID: \"0554f121-8d0b-44b1-b9d2-e5c8d795ee71\") " Oct 30 18:47:20.606896 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.606857 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal" pod "0554f121-8d0b-44b1-b9d2-e5c8d795ee71" (UID: "0554f121-8d0b-44b1-b9d2-e5c8d795ee71"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:20.608665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.608633 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs" pod "0554f121-8d0b-44b1-b9d2-e5c8d795ee71" (UID: "0554f121-8d0b-44b1-b9d2-e5c8d795ee71"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:20.608783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.608682 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert" pod "0554f121-8d0b-44b1-b9d2-e5c8d795ee71" (UID: "0554f121-8d0b-44b1-b9d2-e5c8d795ee71"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:20.608783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.608749 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs" pod "0554f121-8d0b-44b1-b9d2-e5c8d795ee71" (UID: "0554f121-8d0b-44b1-b9d2-e5c8d795ee71"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:20.609047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.609018 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs" pod "0554f121-8d0b-44b1-b9d2-e5c8d795ee71" (UID: "0554f121-8d0b-44b1-b9d2-e5c8d795ee71"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:20.609047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.609033 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt" pod "0554f121-8d0b-44b1-b9d2-e5c8d795ee71" (UID: "0554f121-8d0b-44b1-b9d2-e5c8d795ee71"). InnerVolumeSpecName "kube-api-access-tjbgt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:20.707691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.707657 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjbgt\" (UniqueName: \"kubernetes.io/projected/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-kube-api-access-tjbgt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:20.707691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.707684 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:20.707691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.707694 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:20.707905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.707703 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:20.707905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.707712 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:20.707905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.707720 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/0554f121-8d0b-44b1-b9d2-e5c8d795ee71-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:20.798082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.798052 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:20.808777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:20.808753 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-0"] Oct 30 18:47:21.411635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:21.411606 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0554f121-8d0b-44b1-b9d2-e5c8d795ee71" path="/var/lib/kubelet/pods/0554f121-8d0b-44b1-b9d2-e5c8d795ee71/volumes" Oct 30 18:47:22.641275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:22.641245 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:22.872268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:22.872229 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:23.659337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:23.659303 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:24.562923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.562890 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq"] Oct 30 18:47:24.563200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.563167 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="kube-rbac-proxy" containerID="cri-o://b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" gracePeriod=30 Oct 30 18:47:24.563353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.563295 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="method-gate" containerID="cri-o://50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" gracePeriod=30 Oct 30 18:47:24.563452 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.563280 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="prom-label-proxy" containerID="cri-o://2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" gracePeriod=30 Oct 30 18:47:24.590167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.590142 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x"] Oct 30 18:47:24.590401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.590380 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="kube-rbac-proxy" containerID="cri-o://c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" gracePeriod=30 Oct 30 18:47:24.590527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.590509 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="prom-label-proxy" containerID="cri-o://b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" gracePeriod=30 Oct 30 18:47:24.590580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.590568 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="method-gate" containerID="cri-o://98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" gracePeriod=30 Oct 30 18:47:24.604168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.604147 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn"] Oct 30 18:47:24.604374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.604338 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" podUID="9351828f-758b-463e-a396-87ef93ac3ac9" containerName="korrel8r" containerID="cri-o://89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120" gracePeriod=30 Oct 30 18:47:24.606016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.605993 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj"] Oct 30 18:47:24.606154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.606137 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" podUID="3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" containerName="korrel8r" containerID="cri-o://04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b" gracePeriod=30 Oct 30 18:47:24.608230 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.608207 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:24.909039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.909012 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:47:24.928850 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.928807 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:47:24.937428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.937410 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:47:24.941773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:24.941749 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:25.047972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.047945 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz\" (UniqueName: \"kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz\") pod \"9351828f-758b-463e-a396-87ef93ac3ac9\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " Oct 30 18:47:25.048150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.047990 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h\" (UniqueName: \"kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h\") pod \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " Oct 30 18:47:25.048150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048044 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls\" (UniqueName: \"kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls\") pod \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " Oct 30 18:47:25.048150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048082 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config\" (UniqueName: \"kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config\") pod \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " Oct 30 18:47:25.048150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048118 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8\" (UniqueName: \"kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8\") pod \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " Oct 30 18:47:25.048374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048152 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls\" (UniqueName: \"kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls\") pod \"b46a0366-f077-4f5e-a752-d8e6def549e4\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " Oct 30 18:47:25.048374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048179 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm\") pod \"b46a0366-f077-4f5e-a752-d8e6def549e4\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " Oct 30 18:47:25.048374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048243 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config\") pod \"b46a0366-f077-4f5e-a752-d8e6def549e4\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " Oct 30 18:47:25.048374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048284 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config\" (UniqueName: \"kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config\") pod \"9351828f-758b-463e-a396-87ef93ac3ac9\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " Oct 30 18:47:25.048374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048312 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config\" (UniqueName: \"kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config\") pod \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\" (UID: \"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72\") " Oct 30 18:47:25.048374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048340 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca\") pod \"b46a0366-f077-4f5e-a752-d8e6def549e4\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " Oct 30 18:47:25.048943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048374 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca\") pod \"b46a0366-f077-4f5e-a752-d8e6def549e4\" (UID: \"b46a0366-f077-4f5e-a752-d8e6def549e4\") " Oct 30 18:47:25.048943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048416 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls\" (UniqueName: \"kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls\") pod \"9351828f-758b-463e-a396-87ef93ac3ac9\" (UID: \"9351828f-758b-463e-a396-87ef93ac3ac9\") " Oct 30 18:47:25.048943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048521 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls\" (UniqueName: \"kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls\") pod \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\" (UID: \"8318a332-6b14-4a3b-ae4c-8d1778e1ad31\") " Oct 30 18:47:25.048943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.048796 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config" pod "8318a332-6b14-4a3b-ae4c-8d1778e1ad31" (UID: "8318a332-6b14-4a3b-ae4c-8d1778e1ad31"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.049196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.049052 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config" pod "b46a0366-f077-4f5e-a752-d8e6def549e4" (UID: "b46a0366-f077-4f5e-a752-d8e6def549e4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.049196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.049143 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca" pod "b46a0366-f077-4f5e-a752-d8e6def549e4" (UID: "b46a0366-f077-4f5e-a752-d8e6def549e4"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.049196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.049161 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca" pod "b46a0366-f077-4f5e-a752-d8e6def549e4" (UID: "b46a0366-f077-4f5e-a752-d8e6def549e4"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.050260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.050231 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:47:25.050568 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.050539 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="prometheus" containerID="cri-o://3a806099266f27e5953ce766ca29e851189da29453ca9cca49238034b21b1d91" gracePeriod=600 Oct 30 18:47:25.050823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.050743 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="thanos-sidecar" containerID="cri-o://931f6a54289e74fe2c693edf95ad86bcecaf078a6295c8691e189481d4345e33" gracePeriod=600 Oct 30 18:47:25.050906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.050843 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="config-reloader" containerID="cri-o://f96c7004aeb776c9bee90330eeab33abe7989342f2303dbfa70f1c3771bdd418" gracePeriod=600 Oct 30 18:47:25.050906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.050893 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config" pod "9351828f-758b-463e-a396-87ef93ac3ac9" (UID: "9351828f-758b-463e-a396-87ef93ac3ac9"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.051336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.051311 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config" pod "3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" (UID: "3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.054964 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.054748 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls" pod "3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" (UID: "3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.055626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.055604 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls" pod "9351828f-758b-463e-a396-87ef93ac3ac9" (UID: "9351828f-758b-463e-a396-87ef93ac3ac9"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.058952 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.058913 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h" pod "8318a332-6b14-4a3b-ae4c-8d1778e1ad31" (UID: "8318a332-6b14-4a3b-ae4c-8d1778e1ad31"). InnerVolumeSpecName "kube-api-access-jjk6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.059068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.058993 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:25.059136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.059094 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls" pod "8318a332-6b14-4a3b-ae4c-8d1778e1ad31" (UID: "8318a332-6b14-4a3b-ae4c-8d1778e1ad31"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.059873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.059836 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls" pod "b46a0366-f077-4f5e-a752-d8e6def549e4" (UID: "b46a0366-f077-4f5e-a752-d8e6def549e4"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.060249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.060195 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8" pod "3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" (UID: "3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72"). InnerVolumeSpecName "kube-api-access-q7mw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.060385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.060363 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz" pod "9351828f-758b-463e-a396-87ef93ac3ac9" (UID: "9351828f-758b-463e-a396-87ef93ac3ac9"). InnerVolumeSpecName "kube-api-access-gb7mz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.062733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.062700 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm" pod "b46a0366-f077-4f5e-a752-d8e6def549e4" (UID: "b46a0366-f077-4f5e-a752-d8e6def549e4"). InnerVolumeSpecName "kube-api-access-vqmbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.074665 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.074544 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:47:25.075502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.075349 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="alertmanager" containerID="cri-o://0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64" gracePeriod=120 Oct 30 18:47:25.075612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.075490 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="config-reloader" containerID="cri-o://b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397" gracePeriod=120 Oct 30 18:47:25.150229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150199 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150227 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb7mz\" (UniqueName: \"kubernetes.io/projected/9351828f-758b-463e-a396-87ef93ac3ac9-kube-api-access-gb7mz\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150237 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjk6h\" (UniqueName: \"kubernetes.io/projected/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-kube-api-access-jjk6h\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150246 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150254 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/8318a332-6b14-4a3b-ae4c-8d1778e1ad31-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150262 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7mw8\" (UniqueName: \"kubernetes.io/projected/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-kube-api-access-q7mw8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150270 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/b46a0366-f077-4f5e-a752-d8e6def549e4-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150277 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqmbm\" (UniqueName: \"kubernetes.io/projected/b46a0366-f077-4f5e-a752-d8e6def549e4-kube-api-access-vqmbm\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150285 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150293 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9351828f-758b-463e-a396-87ef93ac3ac9-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150301 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150308 6966 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-client-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150315 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/b46a0366-f077-4f5e-a752-d8e6def549e4-prometheus-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.150451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.150322 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/9351828f-758b-463e-a396-87ef93ac3ac9-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.281187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.281150 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:25.483846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.483818 6966 generic.go:356] "Generic (PLEG): container finished" podID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerID="931f6a54289e74fe2c693edf95ad86bcecaf078a6295c8691e189481d4345e33" exitCode=0 Oct 30 18:47:25.483846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.483844 6966 generic.go:356] "Generic (PLEG): container finished" podID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerID="f96c7004aeb776c9bee90330eeab33abe7989342f2303dbfa70f1c3771bdd418" exitCode=0 Oct 30 18:47:25.484047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.483851 6966 generic.go:356] "Generic (PLEG): container finished" podID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerID="3a806099266f27e5953ce766ca29e851189da29453ca9cca49238034b21b1d91" exitCode=0 Oct 30 18:47:25.484047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.483908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerDied","Data":"931f6a54289e74fe2c693edf95ad86bcecaf078a6295c8691e189481d4345e33"} Oct 30 18:47:25.484047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.483973 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerDied","Data":"f96c7004aeb776c9bee90330eeab33abe7989342f2303dbfa70f1c3771bdd418"} Oct 30 18:47:25.484047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.484003 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerDied","Data":"3a806099266f27e5953ce766ca29e851189da29453ca9cca49238034b21b1d91"} Oct 30 18:47:25.485247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.485224 6966 generic.go:356] "Generic (PLEG): container finished" podID="9351828f-758b-463e-a396-87ef93ac3ac9" containerID="89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120" exitCode=0 Oct 30 18:47:25.485354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.485284 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" Oct 30 18:47:25.485354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.485296 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" event={"ID":"9351828f-758b-463e-a396-87ef93ac3ac9","Type":"ContainerDied","Data":"89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120"} Oct 30 18:47:25.485354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.485342 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn" event={"ID":"9351828f-758b-463e-a396-87ef93ac3ac9","Type":"ContainerDied","Data":"2324a10d7ab2b5eaef9f781acd318f4b27540014d7e1ed17ad02bc21b6924b60"} Oct 30 18:47:25.485518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.485369 6966 scope.go:122] "RemoveContainer" containerID="89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120" Oct 30 18:47:25.487611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487578 6966 generic.go:356] "Generic (PLEG): container finished" podID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerID="2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" exitCode=0 Oct 30 18:47:25.487611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487601 6966 generic.go:356] "Generic (PLEG): container finished" podID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerID="50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" exitCode=0 Oct 30 18:47:25.487611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487611 6966 generic.go:356] "Generic (PLEG): container finished" podID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerID="b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" exitCode=0 Oct 30 18:47:25.487855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487616 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerDied","Data":"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790"} Oct 30 18:47:25.487855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487648 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" Oct 30 18:47:25.487855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487693 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerDied","Data":"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd"} Oct 30 18:47:25.487855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487757 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerDied","Data":"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268"} Oct 30 18:47:25.487855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.487801 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq" event={"ID":"b46a0366-f077-4f5e-a752-d8e6def549e4","Type":"ContainerDied","Data":"3660259f7ece35d9fe4f88a69c51feb6a021d4111cb073e347cf98407118eb5e"} Oct 30 18:47:25.490287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.490265 6966 generic.go:356] "Generic (PLEG): container finished" podID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerID="b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397" exitCode=0 Oct 30 18:47:25.490402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.490343 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerDied","Data":"b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397"} Oct 30 18:47:25.491847 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.491823 6966 generic.go:356] "Generic (PLEG): container finished" podID="3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" containerID="04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b" exitCode=0 Oct 30 18:47:25.491943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.491862 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" Oct 30 18:47:25.491943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.491908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" event={"ID":"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72","Type":"ContainerDied","Data":"04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b"} Oct 30 18:47:25.492067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.491949 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj" event={"ID":"3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72","Type":"ContainerDied","Data":"19941fb739269a2b6f9a7a2be2c9bb44fb734db80b30a3ccfed7989b7d35e57e"} Oct 30 18:47:25.494315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494294 6966 generic.go:356] "Generic (PLEG): container finished" podID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerID="b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" exitCode=0 Oct 30 18:47:25.494416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494315 6966 generic.go:356] "Generic (PLEG): container finished" podID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerID="98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" exitCode=0 Oct 30 18:47:25.494416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494325 6966 generic.go:356] "Generic (PLEG): container finished" podID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerID="c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" exitCode=0 Oct 30 18:47:25.494416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494355 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerDied","Data":"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089"} Oct 30 18:47:25.494416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494365 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" Oct 30 18:47:25.494416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494397 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerDied","Data":"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d"} Oct 30 18:47:25.494712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494427 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerDied","Data":"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc"} Oct 30 18:47:25.494712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.494450 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x" event={"ID":"8318a332-6b14-4a3b-ae4c-8d1778e1ad31","Type":"ContainerDied","Data":"80b0fed09a6f83fb8b83fde37ec665476aa5e769fcb3c406e26bfaeedc6811ee"} Oct 30 18:47:25.495786 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.495766 6966 scope.go:122] "RemoveContainer" containerID="89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120" Oct 30 18:47:25.496128 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.496103 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120\": container with ID starting with 89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120 not found: ID does not exist" containerID="89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120" Oct 30 18:47:25.496279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.496249 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120"} err="failed to get container status \"89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120\": rpc error: code = NotFound desc = could not find container \"89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120\": container with ID starting with 89a57d368aef0429fb8ba306ce9f120f97ca4419b12a503a65ede15a8a2be120 not found: ID does not exist" Oct 30 18:47:25.496279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.496289 6966 scope.go:122] "RemoveContainer" containerID="2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" Oct 30 18:47:25.504200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.504176 6966 scope.go:122] "RemoveContainer" containerID="50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" Oct 30 18:47:25.511285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.511269 6966 scope.go:122] "RemoveContainer" containerID="b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" Oct 30 18:47:25.518394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.518380 6966 scope.go:122] "RemoveContainer" containerID="2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" Oct 30 18:47:25.518652 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.518634 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": container with ID starting with 2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790 not found: ID does not exist" containerID="2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" Oct 30 18:47:25.518724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.518657 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790"} err="failed to get container status \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": rpc error: code = NotFound desc = could not find container \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": container with ID starting with 2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790 not found: ID does not exist" Oct 30 18:47:25.518724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.518672 6966 scope.go:122] "RemoveContainer" containerID="50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" Oct 30 18:47:25.518898 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.518882 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": container with ID starting with 50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd not found: ID does not exist" containerID="50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" Oct 30 18:47:25.518940 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.518903 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd"} err="failed to get container status \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": rpc error: code = NotFound desc = could not find container \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": container with ID starting with 50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd not found: ID does not exist" Oct 30 18:47:25.518940 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.518917 6966 scope.go:122] "RemoveContainer" containerID="b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" Oct 30 18:47:25.519150 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.519136 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": container with ID starting with b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268 not found: ID does not exist" containerID="b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" Oct 30 18:47:25.519194 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519154 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268"} err="failed to get container status \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": rpc error: code = NotFound desc = could not find container \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": container with ID starting with b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268 not found: ID does not exist" Oct 30 18:47:25.519194 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519174 6966 scope.go:122] "RemoveContainer" containerID="2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" Oct 30 18:47:25.519401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519382 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790"} err="failed to get container status \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": rpc error: code = NotFound desc = could not find container \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": container with ID starting with 2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790 not found: ID does not exist" Oct 30 18:47:25.519441 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519402 6966 scope.go:122] "RemoveContainer" containerID="50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" Oct 30 18:47:25.519648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519631 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd"} err="failed to get container status \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": rpc error: code = NotFound desc = could not find container \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": container with ID starting with 50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd not found: ID does not exist" Oct 30 18:47:25.519717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519648 6966 scope.go:122] "RemoveContainer" containerID="b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" Oct 30 18:47:25.519890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519871 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268"} err="failed to get container status \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": rpc error: code = NotFound desc = could not find container \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": container with ID starting with b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268 not found: ID does not exist" Oct 30 18:47:25.519941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.519892 6966 scope.go:122] "RemoveContainer" containerID="2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790" Oct 30 18:47:25.520120 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.520101 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790"} err="failed to get container status \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": rpc error: code = NotFound desc = could not find container \"2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790\": container with ID starting with 2445166256c82a701eb92f8f3df2b68f433a9c3a282d96739d3efe34669f0790 not found: ID does not exist" Oct 30 18:47:25.520165 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.520120 6966 scope.go:122] "RemoveContainer" containerID="50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd" Oct 30 18:47:25.520324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.520305 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd"} err="failed to get container status \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": rpc error: code = NotFound desc = could not find container \"50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd\": container with ID starting with 50f96960bbf0da99295ada8ecb2074104e368982c99a4d8f9ac9a4cd4441dbcd not found: ID does not exist" Oct 30 18:47:25.520397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.520326 6966 scope.go:122] "RemoveContainer" containerID="b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268" Oct 30 18:47:25.520572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.520555 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268"} err="failed to get container status \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": rpc error: code = NotFound desc = could not find container \"b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268\": container with ID starting with b5d4e405e1ec120cf16f954da9657312d791c3eae2e89f5efea168dd80293268 not found: ID does not exist" Oct 30 18:47:25.520652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.520574 6966 scope.go:122] "RemoveContainer" containerID="04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b" Oct 30 18:47:25.527794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.527778 6966 scope.go:122] "RemoveContainer" containerID="04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b" Oct 30 18:47:25.528036 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.528019 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b\": container with ID starting with 04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b not found: ID does not exist" containerID="04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b" Oct 30 18:47:25.528092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.528040 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b"} err="failed to get container status \"04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b\": rpc error: code = NotFound desc = could not find container \"04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b\": container with ID starting with 04d7d2c67cb72e15359dcfe02ee490790d67410083f1774565f70b372444266b not found: ID does not exist" Oct 30 18:47:25.528092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.528052 6966 scope.go:122] "RemoveContainer" containerID="b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" Oct 30 18:47:25.533714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.533684 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq"] Oct 30 18:47:25.535935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.535914 6966 scope.go:122] "RemoveContainer" containerID="98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" Oct 30 18:47:25.538228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.538209 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-82fjq"] Oct 30 18:47:25.543314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.543297 6966 scope.go:122] "RemoveContainer" containerID="c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" Oct 30 18:47:25.550512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.550445 6966 scope.go:122] "RemoveContainer" containerID="b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" Oct 30 18:47:25.550715 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.550697 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": container with ID starting with b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089 not found: ID does not exist" containerID="b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" Oct 30 18:47:25.550789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.550733 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089"} err="failed to get container status \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": rpc error: code = NotFound desc = could not find container \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": container with ID starting with b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089 not found: ID does not exist" Oct 30 18:47:25.550789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.550753 6966 scope.go:122] "RemoveContainer" containerID="98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" Oct 30 18:47:25.550981 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.550962 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": container with ID starting with 98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d not found: ID does not exist" containerID="98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" Oct 30 18:47:25.551028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.550985 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d"} err="failed to get container status \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": rpc error: code = NotFound desc = could not find container \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": container with ID starting with 98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d not found: ID does not exist" Oct 30 18:47:25.551028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551000 6966 scope.go:122] "RemoveContainer" containerID="c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" Oct 30 18:47:25.551249 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:25.551231 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": container with ID starting with c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc not found: ID does not exist" containerID="c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" Oct 30 18:47:25.551293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551253 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc"} err="failed to get container status \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": rpc error: code = NotFound desc = could not find container \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": container with ID starting with c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc not found: ID does not exist" Oct 30 18:47:25.551293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551266 6966 scope.go:122] "RemoveContainer" containerID="b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" Oct 30 18:47:25.551492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551472 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089"} err="failed to get container status \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": rpc error: code = NotFound desc = could not find container \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": container with ID starting with b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089 not found: ID does not exist" Oct 30 18:47:25.551544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551492 6966 scope.go:122] "RemoveContainer" containerID="98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" Oct 30 18:47:25.551736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551715 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d"} err="failed to get container status \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": rpc error: code = NotFound desc = could not find container \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": container with ID starting with 98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d not found: ID does not exist" Oct 30 18:47:25.551813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551738 6966 scope.go:122] "RemoveContainer" containerID="c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" Oct 30 18:47:25.551968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551952 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc"} err="failed to get container status \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": rpc error: code = NotFound desc = could not find container \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": container with ID starting with c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc not found: ID does not exist" Oct 30 18:47:25.552040 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.551969 6966 scope.go:122] "RemoveContainer" containerID="b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089" Oct 30 18:47:25.552195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.552177 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089"} err="failed to get container status \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": rpc error: code = NotFound desc = could not find container \"b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089\": container with ID starting with b525094979c21a36e6caf9090d431d8e88dbf264f9bfad6939d77068cd16e089 not found: ID does not exist" Oct 30 18:47:25.552242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.552196 6966 scope.go:122] "RemoveContainer" containerID="98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d" Oct 30 18:47:25.552401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.552381 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d"} err="failed to get container status \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": rpc error: code = NotFound desc = could not find container \"98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d\": container with ID starting with 98b4efddc9e9c5fb5b4e1c9058608ffd7ba98dc8f33e3c064fad7d8b2a3fdb2d not found: ID does not exist" Oct 30 18:47:25.552451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.552402 6966 scope.go:122] "RemoveContainer" containerID="c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc" Oct 30 18:47:25.552614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.552587 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc"} err="failed to get container status \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": rpc error: code = NotFound desc = could not find container \"c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc\": container with ID starting with c25f442ebdb48f0318d343d4e23f749580b54ef0738292a0cf953ae2b8b3c9bc not found: ID does not exist" Oct 30 18:47:25.568919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.568898 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj"] Oct 30 18:47:25.572977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.572958 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ddpzj"] Oct 30 18:47:25.605366 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.605348 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x"] Oct 30 18:47:25.607664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.607641 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-6rk5x"] Oct 30 18:47:25.610174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.610152 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:25.645546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.645520 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn"] Oct 30 18:47:25.650019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.649976 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-66d6f7f696-vkvqn"] Oct 30 18:47:25.824124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.824102 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:25.828092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.828037 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:25.956660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956634 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956682 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956714 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956754 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956788 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956921 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.956990 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957086 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957142 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957186 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957223 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957256 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957293 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw\") pod \"c2c78515-c331-4bbb-8393-12c31fe0f297\" (UID: \"c2c78515-c331-4bbb-8393-12c31fe0f297\") " Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957345 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.957564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957359 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.957981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957693 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.957981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.957713 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:25.958614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.958584 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:25.959545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.959506 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.960246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960205 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.960337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960272 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.960399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960355 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.960399 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960382 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.960552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960515 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "kube-api-access-f7vrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:25.960925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960907 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:25.961005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.960964 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:25.963944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.963926 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "pvc-c02e8472-bb1a-4974-aa89-86eef715889b". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:47:25.974871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:25.974850 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config" pod "c2c78515-c331-4bbb-8393-12c31fe0f297" (UID: "c2c78515-c331-4bbb-8393-12c31fe0f297"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:26.058454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058424 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058502 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058520 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058535 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058548 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c2c78515-c331-4bbb-8393-12c31fe0f297-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058560 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/c2c78515-c331-4bbb-8393-12c31fe0f297-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058572 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7vrw\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-kube-api-access-f7vrw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058584 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c2c78515-c331-4bbb-8393-12c31fe0f297-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058597 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058610 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.058627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.058623 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/c2c78515-c331-4bbb-8393-12c31fe0f297-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:26.063609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.063583 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:26.071499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.071484 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-c02e8472-bb1a-4974-aa89-86eef715889b" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:47:26.159183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.159124 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-c02e8472-bb1a-4974-aa89-86eef715889b\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-0dedc932cf69fd08e\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-83df9dfad9fd2ac35dca1441af096f35871cdbdee4927aba3c5e3091d60e99b5\"" Oct 30 18:47:26.504391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.504315 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" Oct 30 18:47:26.504391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.504327 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-1" event={"ID":"c2c78515-c331-4bbb-8393-12c31fe0f297","Type":"ContainerDied","Data":"04b8fe722d37cac546592e9a530028e3057b4d1e2f6459cf01f25376e3ec2c21"} Oct 30 18:47:26.504662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.504424 6966 scope.go:122] "RemoveContainer" containerID="931f6a54289e74fe2c693edf95ad86bcecaf078a6295c8691e189481d4345e33" Oct 30 18:47:26.515319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.515301 6966 scope.go:122] "RemoveContainer" containerID="f96c7004aeb776c9bee90330eeab33abe7989342f2303dbfa70f1c3771bdd418" Oct 30 18:47:26.524188 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.524152 6966 scope.go:122] "RemoveContainer" containerID="3a806099266f27e5953ce766ca29e851189da29453ca9cca49238034b21b1d91" Oct 30 18:47:26.534031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.534010 6966 scope.go:122] "RemoveContainer" containerID="494022b41a57ea991283d3e76a2985cdd76dc34b4e4fa16e61a5225840e46f8a" Oct 30 18:47:26.563001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.562969 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:47:26.573283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.573260 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-1"] Oct 30 18:47:26.639583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.639540 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:26.868432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:26.868400 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:27.411961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:27.411930 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" path="/var/lib/kubelet/pods/3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72/volumes" Oct 30 18:47:27.412312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:27.412284 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" path="/var/lib/kubelet/pods/8318a332-6b14-4a3b-ae4c-8d1778e1ad31/volumes" Oct 30 18:47:27.418848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:27.413615 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9351828f-758b-463e-a396-87ef93ac3ac9" path="/var/lib/kubelet/pods/9351828f-758b-463e-a396-87ef93ac3ac9/volumes" Oct 30 18:47:27.418848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:27.414563 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" path="/var/lib/kubelet/pods/b46a0366-f077-4f5e-a752-d8e6def549e4/volumes" Oct 30 18:47:27.418848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:27.415269 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" path="/var/lib/kubelet/pods/c2c78515-c331-4bbb-8393-12c31fe0f297/volumes" Oct 30 18:47:28.712656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:28.712615 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.111:9093/-/ready\": dial tcp 10.134.0.111:9093: connect: connection refused" Oct 30 18:47:33.712624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:33.712583 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.111:9093/-/ready\": dial tcp 10.134.0.111:9093: connect: connection refused" Oct 30 18:47:34.426247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.426206 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:34.649403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649370 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl"] Oct 30 18:47:34.649737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649715 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="prom-label-proxy" Oct 30 18:47:34.649737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649731 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="prom-label-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649746 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="init-config-reloader" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649753 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="init-config-reloader" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649764 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="prometheus" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649772 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="prometheus" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649785 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="prom-label-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649789 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="prom-label-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649799 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="kube-rbac-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649803 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="kube-rbac-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649809 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="config-reloader" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649813 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="config-reloader" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649821 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="kube-rbac-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649827 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="kube-rbac-proxy" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649839 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="method-gate" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649846 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="method-gate" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649855 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0554f121-8d0b-44b1-b9d2-e5c8d795ee71" containerName="otc-container" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649860 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0554f121-8d0b-44b1-b9d2-e5c8d795ee71" containerName="otc-container" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649867 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="thanos-sidecar" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649872 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="thanos-sidecar" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649878 6966 cpu_manager.go:414] "containerMap: removing container" podUID="9351828f-758b-463e-a396-87ef93ac3ac9" containerName="korrel8r" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649882 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9351828f-758b-463e-a396-87ef93ac3ac9" containerName="korrel8r" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649887 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" containerName="korrel8r" Oct 30 18:47:34.649897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649892 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" containerName="korrel8r" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649917 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="method-gate" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649924 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="method-gate" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649988 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="prom-label-proxy" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.649996 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="prometheus" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650028 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="prom-label-proxy" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650037 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="kube-rbac-proxy" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650043 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="config-reloader" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650049 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="kube-rbac-proxy" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650054 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46a0366-f077-4f5e-a752-d8e6def549e4" containerName="method-gate" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650059 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0554f121-8d0b-44b1-b9d2-e5c8d795ee71" containerName="otc-container" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650064 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2c78515-c331-4bbb-8393-12c31fe0f297" containerName="thanos-sidecar" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650072 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="9351828f-758b-463e-a396-87ef93ac3ac9" containerName="korrel8r" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650079 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e7f702b-3fe7-4105-8a6e-e5d2f0d55f72" containerName="korrel8r" Oct 30 18:47:34.651134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.650094 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8318a332-6b14-4a3b-ae4c-8d1778e1ad31" containerName="method-gate" Oct 30 18:47:34.651738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.651545 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:34.653626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.653611 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.677271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.677221 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl"] Oct 30 18:47:34.723114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.723086 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-config\"" Oct 30 18:47:34.723634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.723254 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-dockercfg-fmdm6\"" Oct 30 18:47:34.723725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.723280 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-client-ca\"" Oct 30 18:47:34.723796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.723429 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-prometheus-namespace-proxy-tls\"" Oct 30 18:47:34.723869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.723497 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:47:34.726664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.726637 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5"] Oct 30 18:47:34.731482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.731451 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:34.749203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.749184 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5"] Oct 30 18:47:34.823062 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.823039 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:47:34.823226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.823210 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:47:34.823336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.823323 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-q86pl\"" Oct 30 18:47:34.824652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.824633 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.824718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.824698 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.824755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.824718 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.824809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.824786 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.824849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.824817 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzm29\" (UniqueName: \"kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.870346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.870317 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:34.925805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.925778 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:34.925907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.925817 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpx6\" (UniqueName: \"kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:34.925907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.925894 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.925968 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.925996 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.926018 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.926037 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzm29\" (UniqueName: \"kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.926067 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:34.926640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.926610 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926732 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.926716 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.926774 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.926716 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.928349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.928313 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.948075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.948051 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzm29\" (UniqueName: \"kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29\") pod \"data-science-prometheus-namespace-proxy-69bdfffcb-lhspl\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:34.971298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:34.971278 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:47:35.027256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.027223 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.027385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.027299 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.027385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.027332 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpx6\" (UniqueName: \"kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.028076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.028050 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.030210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.030187 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.049154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.049131 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpx6\" (UniqueName: \"kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6\") pod \"korrel8r-5cbcc89b8f-zhsc5\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.084327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.084298 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:35.270866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.270802 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl"] Oct 30 18:47:35.277235 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:35.277200 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc55512d4_feb2_4d0f_884e_3ceddffda861.slice/crio-ae872fd7a254876e32839c125752e91bd0eb364bbfb9793dd409f651bc1f95ea WatchSource:0}: Error finding container ae872fd7a254876e32839c125752e91bd0eb364bbfb9793dd409f651bc1f95ea: Status 404 returned error can't find the container with id ae872fd7a254876e32839c125752e91bd0eb364bbfb9793dd409f651bc1f95ea Oct 30 18:47:35.306705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.306681 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:35.314663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.314641 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:35.329045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329021 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329147 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329127 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329183 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329224 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329259 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329288 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329325 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x\") pod \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\" (UID: \"45dfc0fe-85a3-43d1-a38b-4db33b4fb760\") " Oct 30 18:47:35.329631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.329582 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:35.331284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.331255 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:47:35.331986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.331961 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:35.332119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.332096 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "kube-api-access-7975x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:35.332598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.332554 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:35.337715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.337694 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:35.344835 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.344805 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config" pod "45dfc0fe-85a3-43d1-a38b-4db33b4fb760" (UID: "45dfc0fe-85a3-43d1-a38b-4db33b4fb760"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:35.344835 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.344828 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:35.430561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430537 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.430675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430563 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.430675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430576 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.430675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430588 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.430675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430600 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.430675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430610 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.430675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.430620 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7975x\" (UniqueName: \"kubernetes.io/projected/45dfc0fe-85a3-43d1-a38b-4db33b4fb760-kube-api-access-7975x\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:47:35.555994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.555966 6966 generic.go:356] "Generic (PLEG): container finished" podID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerID="0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64" exitCode=0 Oct 30 18:47:35.556156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.556022 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:35.556156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.556050 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerDied","Data":"0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64"} Oct 30 18:47:35.556156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.556135 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"45dfc0fe-85a3-43d1-a38b-4db33b4fb760","Type":"ContainerDied","Data":"e8aa9e72ad3599ff707424742d191ef3be70fda6fce700a7b92d09c3a0414050"} Oct 30 18:47:35.556156 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.556151 6966 scope.go:122] "RemoveContainer" containerID="b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397" Oct 30 18:47:35.558415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.558388 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerStarted","Data":"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d"} Oct 30 18:47:35.558547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.558532 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerStarted","Data":"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a"} Oct 30 18:47:35.558630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.558581 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerStarted","Data":"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542"} Oct 30 18:47:35.558630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.558607 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerStarted","Data":"ae872fd7a254876e32839c125752e91bd0eb364bbfb9793dd409f651bc1f95ea"} Oct 30 18:47:35.564567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.564547 6966 scope.go:122] "RemoveContainer" containerID="0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64" Oct 30 18:47:35.572103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.572077 6966 scope.go:122] "RemoveContainer" containerID="44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9" Oct 30 18:47:35.579713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.579697 6966 scope.go:122] "RemoveContainer" containerID="b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397" Oct 30 18:47:35.579944 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:35.579928 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397\": container with ID starting with b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397 not found: ID does not exist" containerID="b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397" Oct 30 18:47:35.580001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.579952 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397"} err="failed to get container status \"b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397\": rpc error: code = NotFound desc = could not find container \"b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397\": container with ID starting with b086bda0289f1732c48d5ce18c10d2982283fb51a7a139e7e3d6e2d1efbce397 not found: ID does not exist" Oct 30 18:47:35.580001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.579970 6966 scope.go:122] "RemoveContainer" containerID="0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64" Oct 30 18:47:35.580196 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:35.580180 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64\": container with ID starting with 0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64 not found: ID does not exist" containerID="0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64" Oct 30 18:47:35.580247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.580201 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64"} err="failed to get container status \"0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64\": rpc error: code = NotFound desc = could not find container \"0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64\": container with ID starting with 0182934d2615d4b4d06075bc78378f9b5f12955fb939e68edbc21a0e63620f64 not found: ID does not exist" Oct 30 18:47:35.580247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.580214 6966 scope.go:122] "RemoveContainer" containerID="44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9" Oct 30 18:47:35.580434 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:35.580417 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9\": container with ID starting with 44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9 not found: ID does not exist" containerID="44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9" Oct 30 18:47:35.580495 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.580441 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9"} err="failed to get container status \"44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9\": rpc error: code = NotFound desc = could not find container \"44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9\": container with ID starting with 44a10ecbbea78dcb7d9464faf7d71ec5b1172f0ca56467d83c977846ed93fac9 not found: ID does not exist" Oct 30 18:47:35.582450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.582431 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:35.606804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.606781 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:47:35.611883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.611864 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:47:35.645794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.645770 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5"] Oct 30 18:47:35.650638 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:35.650609 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe872b97_2748_4e6c_9354_1307e4cfdc67.slice/crio-716c68491fc8dc87fa5493079bc9d8d7c10f8a07bb08e78319071c8d67e7febe WatchSource:0}: Error finding container 716c68491fc8dc87fa5493079bc9d8d7c10f8a07bb08e78319071c8d67e7febe: Status 404 returned error can't find the container with id 716c68491fc8dc87fa5493079bc9d8d7c10f8a07bb08e78319071c8d67e7febe Oct 30 18:47:35.652950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.652908 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" podStartSLOduration=1.6528896579999999 podStartE2EDuration="1.652889658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:35.651362851 +0000 UTC m=+3018.357548209" watchObservedRunningTime="2026-10-30 18:47:35.652889658 +0000 UTC m=+3018.359075014" Oct 30 18:47:35.800789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:35.800755 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.070951 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071395 6966 cpu_manager.go:414] "containerMap: removing container" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="init-config-reloader" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071415 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="init-config-reloader" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071438 6966 cpu_manager.go:414] "containerMap: removing container" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="config-reloader" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071444 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="config-reloader" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071454 6966 cpu_manager.go:414] "containerMap: removing container" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="alertmanager" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071477 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="alertmanager" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071585 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="config-reloader" Oct 30 18:47:36.072414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.071599 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" containerName="alertmanager" Oct 30 18:47:36.080022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.080000 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.095658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.095637 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:47:36.106179 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.106157 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:47:36.136322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.136296 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.136486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.136346 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.136486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.136416 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.136627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.136520 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm8cn\" (UniqueName: \"kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.136627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.136548 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.136712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.136666 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.168557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.168525 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:47:36.168676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.168572 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-57l52\"" Oct 30 18:47:36.168676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.168612 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:47:36.168771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.168742 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:47:36.168824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.168813 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:47:36.191593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.191568 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.237425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.237392 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm8cn\" (UniqueName: \"kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.237425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.237426 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.237672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.237488 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.237672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.237515 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.237672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.237556 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.237672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.237585 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.238383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.238353 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.240023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.239994 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.240151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.240041 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.240151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.240084 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.240151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.240096 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.259983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.259938 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm8cn\" (UniqueName: \"kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn\") pod \"data-science-collector-collector-1\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.394313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.394237 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:47:36.414021 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.413992 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.565285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.565250 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" event={"ID":"fe872b97-2748-4e6c-9354-1307e4cfdc67","Type":"ContainerStarted","Data":"6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e"} Oct 30 18:47:36.565435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.565321 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:36.565435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.565341 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" event={"ID":"fe872b97-2748-4e6c-9354-1307e4cfdc67","Type":"ContainerStarted","Data":"716c68491fc8dc87fa5493079bc9d8d7c10f8a07bb08e78319071c8d67e7febe"} Oct 30 18:47:36.605114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.605068 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" podStartSLOduration=2.605054773 podStartE2EDuration="2.605054773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:36.603641317 +0000 UTC m=+3019.309826674" watchObservedRunningTime="2026-10-30 18:47:36.605054773 +0000 UTC m=+3019.311240129" Oct 30 18:47:36.643569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.643543 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:36.697413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.697356 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:47:36.702044 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:36.702023 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f06aa0_5c25_40a9_b18d_f58ab7566349.slice/crio-e3adf67cbcd2189c1c7ff0d476b2106944b1b2191957118df823ff2dc7fe3c41 WatchSource:0}: Error finding container e3adf67cbcd2189c1c7ff0d476b2106944b1b2191957118df823ff2dc7fe3c41: Status 404 returned error can't find the container with id e3adf67cbcd2189c1c7ff0d476b2106944b1b2191957118df823ff2dc7fe3c41 Oct 30 18:47:36.703930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.703899 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:47:36.704028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:36.703964 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:47:37.414456 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:37.413988 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45dfc0fe-85a3-43d1-a38b-4db33b4fb760" path="/var/lib/kubelet/pods/45dfc0fe-85a3-43d1-a38b-4db33b4fb760/volumes" Oct 30 18:47:37.573959 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:37.573914 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73f06aa0-5c25-40a9-b18d-f58ab7566349","Type":"ContainerStarted","Data":"718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9"} Oct 30 18:47:37.574127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:37.574000 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73f06aa0-5c25-40a9-b18d-f58ab7566349","Type":"ContainerStarted","Data":"e3adf67cbcd2189c1c7ff0d476b2106944b1b2191957118df823ff2dc7fe3c41"} Oct 30 18:47:37.626692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:37.626646 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.6266329210000001 podStartE2EDuration="1.626632921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:37.615062049 +0000 UTC m=+3020.321247408" watchObservedRunningTime="2026-10-30 18:47:37.626632921 +0000 UTC m=+3020.332818345" Oct 30 18:47:43.247373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.247336 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:43.786396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.786360 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:47:43.791356 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.791333 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:43.806937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.806906 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:47:43.818355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.816336 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:47:43.821075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.821050 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:47:43.821242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.821184 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:43.828450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.828429 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:47:43.887300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887271 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:47:43.887300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887282 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:47:43.887300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887275 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:47:43.887300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887299 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:47:43.887610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887309 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:47:43.887610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887432 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:47:43.887610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887472 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:47:43.887610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887570 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:47:43.887610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887585 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:47:43.887610 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887590 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-98zpq\"" Oct 30 18:47:43.887818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887630 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:47:43.887818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.887661 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-b9xnn\"" Oct 30 18:47:43.906911 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.906891 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:47:43.907012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:43.906928 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:47:44.004367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004336 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.004540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004369 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.004540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004397 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004517 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004565 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpsvr\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004609 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004631 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004688 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004724 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004748 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.004997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004790 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.004997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004810 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.004997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004827 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.004997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004850 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004868 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.004997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.004985 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.005322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.005017 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.005322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.005056 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.005322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.005075 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h8r6\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106360 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106394 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106426 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106444 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h8r6\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106570 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106604 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106643 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106670 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106708 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpsvr\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106736 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106768 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106797 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106837 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106860 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106895 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106927 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.106968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.106959 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.107363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.107001 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.107363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.107030 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.107363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.107096 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.107557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.107437 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.108016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.107988 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.109219 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.109191 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.109806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.109756 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.110131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.110104 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.110779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.110335 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.110779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.110671 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.110779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.110725 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.111147 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.111121 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.111480 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.111436 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.111973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.111951 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.112063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.112007 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.112412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.112388 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.112534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.112410 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.112534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.112489 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.113444 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.113428 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.129310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.129285 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h8r6\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.130878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.130858 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpsvr\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:44.141676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.141656 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:44.455438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.455353 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:47:44.459498 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:44.459445 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod008637e6_bef8_4abf_8849_b5cf9a3cb6d0.slice/crio-75a5c34b578e298c3ecf785d3df9e8d7bec26b6752755ed2a021a70371432a81 WatchSource:0}: Error finding container 75a5c34b578e298c3ecf785d3df9e8d7bec26b6752755ed2a021a70371432a81: Status 404 returned error can't find the container with id 75a5c34b578e298c3ecf785d3df9e8d7bec26b6752755ed2a021a70371432a81 Oct 30 18:47:44.609709 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.609682 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerStarted","Data":"75a5c34b578e298c3ecf785d3df9e8d7bec26b6752755ed2a021a70371432a81"} Oct 30 18:47:44.632552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:44.632524 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:45.616816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:45.616779 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerStarted","Data":"4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c"} Oct 30 18:47:47.199600 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:47:47.199564 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:47:47.576651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:47.576623 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:47:49.638886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:49.638856 6966 generic.go:356] "Generic (PLEG): container finished" podID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerID="4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c" exitCode=0 Oct 30 18:47:49.639285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:49.638933 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerDied","Data":"4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c"} Oct 30 18:47:49.956002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:49.955921 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:49.983373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:49.983341 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.056625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.056598 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.056764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.056743 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.070408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.070385 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.170870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.170838 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.196159 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.196136 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.428293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.428267 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:47:50.647276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.647243 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerStarted","Data":"3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5"} Oct 30 18:47:50.647638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.647306 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerStarted","Data":"0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4"} Oct 30 18:47:50.647638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.647499 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:50.703722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.703674 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.917935408 podStartE2EDuration="7.703660046s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.785724638s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:50.700951311 +0000 UTC m=+3033.407136667" watchObservedRunningTime="2026-10-30 18:47:50.703660046 +0000 UTC m=+3033.409845402" Oct 30 18:47:50.760402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:50.760371 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:47:50.765045 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:47:50.765018 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05d1fed0_0d94_4859_b4bb_7e53696e21f4.slice/crio-c9029b5f8b2edcfb768efedbf0d5820022256f7456577467d5282a14cf312041 WatchSource:0}: Error finding container c9029b5f8b2edcfb768efedbf0d5820022256f7456577467d5282a14cf312041: Status 404 returned error can't find the container with id c9029b5f8b2edcfb768efedbf0d5820022256f7456577467d5282a14cf312041 Oct 30 18:47:51.654297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:51.654265 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerStarted","Data":"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb"} Oct 30 18:47:51.654688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:51.654331 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerStarted","Data":"c9029b5f8b2edcfb768efedbf0d5820022256f7456577467d5282a14cf312041"} Oct 30 18:47:52.436041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:52.436001 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:52.655647 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:52.655603 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:55.676364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:55.676331 6966 generic.go:356] "Generic (PLEG): container finished" podID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerID="6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb" exitCode=0 Oct 30 18:47:55.676771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:55.676402 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerDied","Data":"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb"} Oct 30 18:47:56.656141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:56.656111 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:47:56.684530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:56.684490 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerStarted","Data":"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8"} Oct 30 18:47:56.684969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:56.684564 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerStarted","Data":"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60"} Oct 30 18:47:56.684969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:56.684615 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerStarted","Data":"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f"} Oct 30 18:47:56.753055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:56.752996 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.165899699 podStartE2EDuration="13.752981667s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.587081968s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:47:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:56.751846973 +0000 UTC m=+3039.458032331" watchObservedRunningTime="2026-10-30 18:47:56.752981667 +0000 UTC m=+3039.459167023" Oct 30 18:47:58.259039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:58.259005 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:58.481822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:58.481780 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:58.789472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:58.789421 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:47:59.133402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:47:59.133367 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:00.429029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:00.428986 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:48:05.428927 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:05.428895 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:48:05.434416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:05.434395 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:48:05.738167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:05.738099 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:48:06.135593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:06.135556 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.088584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:12.088527 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.325308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:12.325271 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:12.543329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:12.543242 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:17.200067 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:17.200029 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:48:17.862360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:17.862321 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.085646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:18.085603 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.683534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:18.683500 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:18.954282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:18.954198 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:19.208864 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:19.208785 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:22.786295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:22.786261 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:23.595559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:23.595527 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:24.155365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:24.155329 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.285520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:25.285485 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.585003 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:25.584968 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:25.806920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:25.806886 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.323928 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:29.323853 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.585870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:29.585783 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:29.891885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:29.891811 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:30.126371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:30.126335 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.464757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:35.464718 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:35.752441 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:35.752367 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:36.313114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:36.313064 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:36.547367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:36.547337 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:41.143762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:41.143727 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:41.384035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:41.384001 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:42.615561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:42.615529 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:42.845408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:42.845374 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:46.823264 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:46.823222 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:47.047789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:47.047756 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:47.199178 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:47.199106 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:48:47.422054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:47.422012 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:52.611254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:52.611220 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:52.826789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:52.826754 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:54.048092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.048055 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl"] Oct 30 18:48:54.048589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.048356 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="kube-rbac-proxy" containerID="cri-o://6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" gracePeriod=30 Oct 30 18:48:54.048589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.048429 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="prom-label-proxy" containerID="cri-o://2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" gracePeriod=30 Oct 30 18:48:54.049191 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.048525 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="method-gate" containerID="cri-o://7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" gracePeriod=30 Oct 30 18:48:54.084367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.084335 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5"] Oct 30 18:48:54.084577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.084556 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" podUID="fe872b97-2748-4e6c-9354-1307e4cfdc67" containerName="korrel8r" containerID="cri-o://6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e" gracePeriod=30 Oct 30 18:48:54.333681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.333655 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:48:54.363304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.363191 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488539 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls\" (UniqueName: \"kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls\") pod \"fe872b97-2748-4e6c-9354-1307e4cfdc67\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488597 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config\" (UniqueName: \"kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config\") pod \"fe872b97-2748-4e6c-9354-1307e4cfdc67\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488647 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls\" (UniqueName: \"kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls\") pod \"c55512d4-feb2-4d0f-884e-3ceddffda861\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488686 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca\") pod \"c55512d4-feb2-4d0f-884e-3ceddffda861\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488723 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config\") pod \"c55512d4-feb2-4d0f-884e-3ceddffda861\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488766 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29\" (UniqueName: \"kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29\") pod \"c55512d4-feb2-4d0f-884e-3ceddffda861\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488804 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca\") pod \"c55512d4-feb2-4d0f-884e-3ceddffda861\" (UID: \"c55512d4-feb2-4d0f-884e-3ceddffda861\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.488846 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6\" (UniqueName: \"kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6\") pod \"fe872b97-2748-4e6c-9354-1307e4cfdc67\" (UID: \"fe872b97-2748-4e6c-9354-1307e4cfdc67\") " Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.490155 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca" pod "c55512d4-feb2-4d0f-884e-3ceddffda861" (UID: "c55512d4-feb2-4d0f-884e-3ceddffda861"). InnerVolumeSpecName "prometheus-tls-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.490338 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca" pod "c55512d4-feb2-4d0f-884e-3ceddffda861" (UID: "c55512d4-feb2-4d0f-884e-3ceddffda861"). InnerVolumeSpecName "client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.490542 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config" pod "c55512d4-feb2-4d0f-884e-3ceddffda861" (UID: "c55512d4-feb2-4d0f-884e-3ceddffda861"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.491490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.491016 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config" pod "fe872b97-2748-4e6c-9354-1307e4cfdc67" (UID: "fe872b97-2748-4e6c-9354-1307e4cfdc67"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.492888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.492862 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls" pod "fe872b97-2748-4e6c-9354-1307e4cfdc67" (UID: "fe872b97-2748-4e6c-9354-1307e4cfdc67"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.494308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.494269 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls" pod "c55512d4-feb2-4d0f-884e-3ceddffda861" (UID: "c55512d4-feb2-4d0f-884e-3ceddffda861"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.494424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.494401 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29" pod "c55512d4-feb2-4d0f-884e-3ceddffda861" (UID: "c55512d4-feb2-4d0f-884e-3ceddffda861"). InnerVolumeSpecName "kube-api-access-wzm29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:54.494424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.494417 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6" pod "fe872b97-2748-4e6c-9354-1307e4cfdc67" (UID: "fe872b97-2748-4e6c-9354-1307e4cfdc67"). InnerVolumeSpecName "kube-api-access-5mpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:54.515304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.515275 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:48:54.515579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.515540 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="alertmanager" containerID="cri-o://0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4" gracePeriod=120 Oct 30 18:48:54.515883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.515857 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="config-reloader" containerID="cri-o://3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5" gracePeriod=120 Oct 30 18:48:54.526715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.526689 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:48:54.526969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.526940 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="thanos-sidecar" containerID="cri-o://dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" gracePeriod=600 Oct 30 18:48:54.526969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.526956 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="config-reloader" containerID="cri-o://08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" gracePeriod=600 Oct 30 18:48:54.527167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.526923 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="prometheus" containerID="cri-o://2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" gracePeriod=600 Oct 30 18:48:54.533907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.533882 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:54.590446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590391 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/fe872b97-2748-4e6c-9354-1307e4cfdc67-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590415 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fe872b97-2748-4e6c-9354-1307e4cfdc67-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590423 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/c55512d4-feb2-4d0f-884e-3ceddffda861-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590432 6966 reconciler_common.go:299] "Volume detached for volume \"client-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-client-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590441 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590449 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzm29\" (UniqueName: \"kubernetes.io/projected/c55512d4-feb2-4d0f-884e-3ceddffda861-kube-api-access-wzm29\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590471 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-tls-ca\" (UniqueName: \"kubernetes.io/configmap/c55512d4-feb2-4d0f-884e-3ceddffda861-prometheus-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.590724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.590486 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpx6\" (UniqueName: \"kubernetes.io/projected/fe872b97-2748-4e6c-9354-1307e4cfdc67-kube-api-access-5mpx6\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.750114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.750083 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:54.851018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.850993 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:48:54.972132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.972083 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:54.994131 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994108 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994154 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994177 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994197 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994278 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994320 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994371 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994401 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994427 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994482 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994549 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.994752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994562 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994634 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.994752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.994716 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file\") pod \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\" (UID: \"05d1fed0-0d94-4859-b4bb-7e53696e21f4\") " Oct 30 18:48:54.995036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.995022 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:54.995555 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.995528 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.995812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.995703 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:54.997018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.996953 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.997160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.997063 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.997784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.997744 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:54.998397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.998361 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.998685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.998650 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:54.998812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.998781 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:54.999752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.999699 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "kube-api-access-lpsvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:54.999752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:54.999734 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.000569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.000550 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:48:55.001494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001450 6966 generic.go:356] "Generic (PLEG): container finished" podID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerID="2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" exitCode=0 Oct 30 18:48:55.001596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001497 6966 generic.go:356] "Generic (PLEG): container finished" podID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerID="7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" exitCode=0 Oct 30 18:48:55.001596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001509 6966 generic.go:356] "Generic (PLEG): container finished" podID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerID="6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" exitCode=0 Oct 30 18:48:55.001596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001509 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerDied","Data":"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d"} Oct 30 18:48:55.001596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001540 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" Oct 30 18:48:55.001596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001594 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerDied","Data":"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a"} Oct 30 18:48:55.001871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001631 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerDied","Data":"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542"} Oct 30 18:48:55.001871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001669 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl" event={"ID":"c55512d4-feb2-4d0f-884e-3ceddffda861","Type":"ContainerDied","Data":"ae872fd7a254876e32839c125752e91bd0eb364bbfb9793dd409f651bc1f95ea"} Oct 30 18:48:55.001871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.001701 6966 scope.go:122] "RemoveContainer" containerID="2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" Oct 30 18:48:55.005570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.005543 6966 generic.go:356] "Generic (PLEG): container finished" podID="fe872b97-2748-4e6c-9354-1307e4cfdc67" containerID="6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e" exitCode=0 Oct 30 18:48:55.005677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.005612 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" Oct 30 18:48:55.005677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.005636 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" event={"ID":"fe872b97-2748-4e6c-9354-1307e4cfdc67","Type":"ContainerDied","Data":"6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e"} Oct 30 18:48:55.005794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.005688 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5" event={"ID":"fe872b97-2748-4e6c-9354-1307e4cfdc67","Type":"ContainerDied","Data":"716c68491fc8dc87fa5493079bc9d8d7c10f8a07bb08e78319071c8d67e7febe"} Oct 30 18:48:55.011388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011364 6966 generic.go:356] "Generic (PLEG): container finished" podID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerID="dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" exitCode=0 Oct 30 18:48:55.011388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011387 6966 generic.go:356] "Generic (PLEG): container finished" podID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerID="08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" exitCode=0 Oct 30 18:48:55.011568 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011396 6966 generic.go:356] "Generic (PLEG): container finished" podID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerID="2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" exitCode=0 Oct 30 18:48:55.011637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011572 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerDied","Data":"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8"} Oct 30 18:48:55.011637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011607 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:48:55.011753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011635 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerDied","Data":"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60"} Oct 30 18:48:55.011753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011679 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerDied","Data":"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f"} Oct 30 18:48:55.011753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.011716 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"05d1fed0-0d94-4859-b4bb-7e53696e21f4","Type":"ContainerDied","Data":"c9029b5f8b2edcfb768efedbf0d5820022256f7456577467d5282a14cf312041"} Oct 30 18:48:55.012200 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.012178 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config" pod "05d1fed0-0d94-4859-b4bb-7e53696e21f4" (UID: "05d1fed0-0d94-4859-b4bb-7e53696e21f4"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.014090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.014071 6966 generic.go:356] "Generic (PLEG): container finished" podID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerID="3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5" exitCode=0 Oct 30 18:48:55.014169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.014093 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerDied","Data":"3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5"} Oct 30 18:48:55.014550 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.014535 6966 scope.go:122] "RemoveContainer" containerID="7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" Oct 30 18:48:55.022488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.022455 6966 scope.go:122] "RemoveContainer" containerID="6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" Oct 30 18:48:55.029803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.029787 6966 scope.go:122] "RemoveContainer" containerID="2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" Oct 30 18:48:55.030056 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.030036 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": container with ID starting with 2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d not found: ID does not exist" containerID="2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" Oct 30 18:48:55.030108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030064 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d"} err="failed to get container status \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": rpc error: code = NotFound desc = could not find container \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": container with ID starting with 2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d not found: ID does not exist" Oct 30 18:48:55.030108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030079 6966 scope.go:122] "RemoveContainer" containerID="7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" Oct 30 18:48:55.030337 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.030319 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": container with ID starting with 7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a not found: ID does not exist" containerID="7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" Oct 30 18:48:55.030376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030343 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a"} err="failed to get container status \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": rpc error: code = NotFound desc = could not find container \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": container with ID starting with 7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a not found: ID does not exist" Oct 30 18:48:55.030376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030358 6966 scope.go:122] "RemoveContainer" containerID="6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" Oct 30 18:48:55.030598 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.030582 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": container with ID starting with 6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542 not found: ID does not exist" containerID="6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" Oct 30 18:48:55.030642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030604 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542"} err="failed to get container status \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": rpc error: code = NotFound desc = could not find container \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": container with ID starting with 6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542 not found: ID does not exist" Oct 30 18:48:55.030642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030619 6966 scope.go:122] "RemoveContainer" containerID="2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" Oct 30 18:48:55.030815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030800 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d"} err="failed to get container status \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": rpc error: code = NotFound desc = could not find container \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": container with ID starting with 2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d not found: ID does not exist" Oct 30 18:48:55.030858 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.030815 6966 scope.go:122] "RemoveContainer" containerID="7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" Oct 30 18:48:55.031020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031002 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a"} err="failed to get container status \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": rpc error: code = NotFound desc = could not find container \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": container with ID starting with 7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a not found: ID does not exist" Oct 30 18:48:55.031073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031021 6966 scope.go:122] "RemoveContainer" containerID="6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" Oct 30 18:48:55.031267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031250 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542"} err="failed to get container status \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": rpc error: code = NotFound desc = could not find container \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": container with ID starting with 6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542 not found: ID does not exist" Oct 30 18:48:55.031311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031269 6966 scope.go:122] "RemoveContainer" containerID="2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d" Oct 30 18:48:55.031506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031488 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d"} err="failed to get container status \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": rpc error: code = NotFound desc = could not find container \"2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d\": container with ID starting with 2bc024697a442ace34c75e45b273f65019bc4a0e7190f7d96d96aa9dec7dfe0d not found: ID does not exist" Oct 30 18:48:55.031558 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031506 6966 scope.go:122] "RemoveContainer" containerID="7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a" Oct 30 18:48:55.031862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031845 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a"} err="failed to get container status \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": rpc error: code = NotFound desc = could not find container \"7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a\": container with ID starting with 7c05028b8636b93999b83a8a0207fd52c883d74fe8ee0fb374e4d86e0e87d81a not found: ID does not exist" Oct 30 18:48:55.031902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.031862 6966 scope.go:122] "RemoveContainer" containerID="6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542" Oct 30 18:48:55.032124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.032107 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542"} err="failed to get container status \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": rpc error: code = NotFound desc = could not find container \"6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542\": container with ID starting with 6158c1ce52c55d9f01b6b74ceca18f22bcdcb3b140dfdafe554ccc9b94d25542 not found: ID does not exist" Oct 30 18:48:55.032169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.032124 6966 scope.go:122] "RemoveContainer" containerID="6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e" Oct 30 18:48:55.039380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.039364 6966 scope.go:122] "RemoveContainer" containerID="6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e" Oct 30 18:48:55.039683 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.039666 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e\": container with ID starting with 6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e not found: ID does not exist" containerID="6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e" Oct 30 18:48:55.039747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.039689 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e"} err="failed to get container status \"6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e\": rpc error: code = NotFound desc = could not find container \"6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e\": container with ID starting with 6757cd71bfd8cc6e052c217a76ecc91b159e12efb20545bc6a40091ab910110e not found: ID does not exist" Oct 30 18:48:55.039747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.039707 6966 scope.go:122] "RemoveContainer" containerID="dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" Oct 30 18:48:55.046924 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.046902 6966 scope.go:122] "RemoveContainer" containerID="08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" Oct 30 18:48:55.053375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.053356 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl"] Oct 30 18:48:55.055090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.054946 6966 scope.go:122] "RemoveContainer" containerID="2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" Oct 30 18:48:55.059308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.059289 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-prometheus-namespace-proxy-69bdfffcb-lhspl"] Oct 30 18:48:55.086405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.086378 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:48:55.086680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.086649 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="73f06aa0-5c25-40a9-b18d-f58ab7566349" containerName="otc-container" containerID="cri-o://718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9" gracePeriod=30 Oct 30 18:48:55.088714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.088691 6966 scope.go:122] "RemoveContainer" containerID="6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095694 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095723 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095740 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095753 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095787 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095802 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095817 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095833 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/05d1fed0-0d94-4859-b4bb-7e53696e21f4-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095846 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpsvr\" (UniqueName: \"kubernetes.io/projected/05d1fed0-0d94-4859-b4bb-7e53696e21f4-kube-api-access-lpsvr\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095861 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095876 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/05d1fed0-0d94-4859-b4bb-7e53696e21f4-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.095893 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/05d1fed0-0d94-4859-b4bb-7e53696e21f4-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.098505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.098070 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5"] Oct 30 18:48:55.100821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.099977 6966 scope.go:122] "RemoveContainer" containerID="dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" Oct 30 18:48:55.100821 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.100572 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": container with ID starting with dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8 not found: ID does not exist" containerID="dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" Oct 30 18:48:55.100821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.100598 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8"} err="failed to get container status \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": rpc error: code = NotFound desc = could not find container \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": container with ID starting with dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8 not found: ID does not exist" Oct 30 18:48:55.100821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.100619 6966 scope.go:122] "RemoveContainer" containerID="08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" Oct 30 18:48:55.101124 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.100857 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": container with ID starting with 08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60 not found: ID does not exist" containerID="08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" Oct 30 18:48:55.101124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.100878 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60"} err="failed to get container status \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": rpc error: code = NotFound desc = could not find container \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": container with ID starting with 08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60 not found: ID does not exist" Oct 30 18:48:55.101124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.100893 6966 scope.go:122] "RemoveContainer" containerID="2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" Oct 30 18:48:55.101836 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.101665 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": container with ID starting with 2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f not found: ID does not exist" containerID="2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" Oct 30 18:48:55.101836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.101700 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f"} err="failed to get container status \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": rpc error: code = NotFound desc = could not find container \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": container with ID starting with 2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f not found: ID does not exist" Oct 30 18:48:55.101836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.101731 6966 scope.go:122] "RemoveContainer" containerID="6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb" Oct 30 18:48:55.102235 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:55.102106 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": container with ID starting with 6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb not found: ID does not exist" containerID="6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb" Oct 30 18:48:55.102235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.102137 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb"} err="failed to get container status \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": rpc error: code = NotFound desc = could not find container \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": container with ID starting with 6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb not found: ID does not exist" Oct 30 18:48:55.102235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.102159 6966 scope.go:122] "RemoveContainer" containerID="dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" Oct 30 18:48:55.102574 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.102549 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8"} err="failed to get container status \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": rpc error: code = NotFound desc = could not find container \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": container with ID starting with dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8 not found: ID does not exist" Oct 30 18:48:55.102781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.102576 6966 scope.go:122] "RemoveContainer" containerID="08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" Oct 30 18:48:55.103007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.102982 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60"} err="failed to get container status \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": rpc error: code = NotFound desc = could not find container \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": container with ID starting with 08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60 not found: ID does not exist" Oct 30 18:48:55.103067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.103010 6966 scope.go:122] "RemoveContainer" containerID="2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" Oct 30 18:48:55.103401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.103366 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f"} err="failed to get container status \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": rpc error: code = NotFound desc = could not find container \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": container with ID starting with 2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f not found: ID does not exist" Oct 30 18:48:55.103401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.103391 6966 scope.go:122] "RemoveContainer" containerID="6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb" Oct 30 18:48:55.103749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.103728 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb"} err="failed to get container status \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": rpc error: code = NotFound desc = could not find container \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": container with ID starting with 6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb not found: ID does not exist" Oct 30 18:48:55.103749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.103748 6966 scope.go:122] "RemoveContainer" containerID="dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8" Oct 30 18:48:55.104055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.104030 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8"} err="failed to get container status \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": rpc error: code = NotFound desc = could not find container \"dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8\": container with ID starting with dd5b8e0bf446493f51536c1a73c7e36b8891960be7c6a7a4d08ab56fa16d1ee8 not found: ID does not exist" Oct 30 18:48:55.104266 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.104056 6966 scope.go:122] "RemoveContainer" containerID="08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60" Oct 30 18:48:55.104394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.104351 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60"} err="failed to get container status \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": rpc error: code = NotFound desc = could not find container \"08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60\": container with ID starting with 08c358038aec7ae5a3f1b7c58af1ea8b07ca7b0f40207fc5d7a9e0413f02af60 not found: ID does not exist" Oct 30 18:48:55.104394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.104375 6966 scope.go:122] "RemoveContainer" containerID="2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f" Oct 30 18:48:55.105448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.105341 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f"} err="failed to get container status \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": rpc error: code = NotFound desc = could not find container \"2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f\": container with ID starting with 2992f8e46b295ba41b379ad9eca3a097118053bcdeb5e9fa3f666efbf71b3d5f not found: ID does not exist" Oct 30 18:48:55.105448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.105367 6966 scope.go:122] "RemoveContainer" containerID="6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb" Oct 30 18:48:55.108871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.108830 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-zhsc5"] Oct 30 18:48:55.109722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.109673 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb"} err="failed to get container status \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": rpc error: code = NotFound desc = could not find container \"6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb\": container with ID starting with 6b04fe94db864c8b1d7757a6ebda8894b79d21455c3198070a7fed26fe6938cb not found: ID does not exist" Oct 30 18:48:55.115313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.115295 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:48:55.196937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.196905 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 18:48:55.238674 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.238648 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:55.356157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.356109 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:48:55.359683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.359663 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:48:55.366077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.366052 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:48:55.384996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.384976 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:48:55.411769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.411746 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" path="/var/lib/kubelet/pods/05d1fed0-0d94-4859-b4bb-7e53696e21f4/volumes" Oct 30 18:48:55.412293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.412277 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" path="/var/lib/kubelet/pods/c55512d4-feb2-4d0f-884e-3ceddffda861/volumes" Oct 30 18:48:55.413067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.413049 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe872b97-2748-4e6c-9354-1307e4cfdc67" path="/var/lib/kubelet/pods/fe872b97-2748-4e6c-9354-1307e4cfdc67/volumes" Oct 30 18:48:55.485838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.485800 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:55.499637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499607 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal\" (UniqueName: \"kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal\") pod \"73f06aa0-5c25-40a9-b18d-f58ab7566349\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " Oct 30 18:48:55.499801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499651 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs\") pod \"73f06aa0-5c25-40a9-b18d-f58ab7566349\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " Oct 30 18:48:55.499801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499684 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.499801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499724 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs\") pod \"73f06aa0-5c25-40a9-b18d-f58ab7566349\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " Oct 30 18:48:55.499801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499751 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.499801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499782 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.500074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499821 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.500074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.499942 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn\" (UniqueName: \"kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn\") pod \"73f06aa0-5c25-40a9-b18d-f58ab7566349\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " Oct 30 18:48:55.500074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500064 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs\") pod \"73f06aa0-5c25-40a9-b18d-f58ab7566349\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " Oct 30 18:48:55.500243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500068 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal" pod "73f06aa0-5c25-40a9-b18d-f58ab7566349" (UID: "73f06aa0-5c25-40a9-b18d-f58ab7566349"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:55.500243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500095 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.500243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500138 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert\") pod \"73f06aa0-5c25-40a9-b18d-f58ab7566349\" (UID: \"73f06aa0-5c25-40a9-b18d-f58ab7566349\") " Oct 30 18:48:55.500243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500173 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.500243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500212 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db\") pod \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\" (UID: \"008637e6-bef8-4abf-8849-b5cf9a3cb6d0\") " Oct 30 18:48:55.500571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.500547 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/73f06aa0-5c25-40a9-b18d-f58ab7566349-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.501326 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.501301 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:55.502714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.502682 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn" pod "73f06aa0-5c25-40a9-b18d-f58ab7566349" (UID: "73f06aa0-5c25-40a9-b18d-f58ab7566349"). InnerVolumeSpecName "kube-api-access-fm8cn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:55.502827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.502724 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "kube-api-access-4h8r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:55.502897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.502833 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs" pod "73f06aa0-5c25-40a9-b18d-f58ab7566349" (UID: "73f06aa0-5c25-40a9-b18d-f58ab7566349"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.504328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.504293 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.504328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.504309 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:55.504938 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.504913 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:55.505395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.505369 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert" pod "73f06aa0-5c25-40a9-b18d-f58ab7566349" (UID: "73f06aa0-5c25-40a9-b18d-f58ab7566349"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.505946 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.505921 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs" pod "73f06aa0-5c25-40a9-b18d-f58ab7566349" (UID: "73f06aa0-5c25-40a9-b18d-f58ab7566349"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.507639 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.507606 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs" pod "73f06aa0-5c25-40a9-b18d-f58ab7566349" (UID: "73f06aa0-5c25-40a9-b18d-f58ab7566349"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.510782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.510761 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.516161 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.516139 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config" pod "008637e6-bef8-4abf-8849-b5cf9a3cb6d0" (UID: "008637e6-bef8-4abf-8849-b5cf9a3cb6d0"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:55.601608 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601578 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601608 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601603 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601620 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601631 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h8r6\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-kube-api-access-4h8r6\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601640 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601648 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601657 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601665 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601673 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm8cn\" (UniqueName: \"kubernetes.io/projected/73f06aa0-5c25-40a9-b18d-f58ab7566349-kube-api-access-fm8cn\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601682 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601689 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/008637e6-bef8-4abf-8849-b5cf9a3cb6d0-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:55.601783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:55.601698 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/73f06aa0-5c25-40a9-b18d-f58ab7566349-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:48:56.023257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.023226 6966 generic.go:356] "Generic (PLEG): container finished" podID="73f06aa0-5c25-40a9-b18d-f58ab7566349" containerID="718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9" exitCode=0 Oct 30 18:48:56.023454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.023269 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:48:56.023454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.023309 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73f06aa0-5c25-40a9-b18d-f58ab7566349","Type":"ContainerDied","Data":"718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9"} Oct 30 18:48:56.023454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.023376 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"73f06aa0-5c25-40a9-b18d-f58ab7566349","Type":"ContainerDied","Data":"e3adf67cbcd2189c1c7ff0d476b2106944b1b2191957118df823ff2dc7fe3c41"} Oct 30 18:48:56.023454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.023406 6966 scope.go:122] "RemoveContainer" containerID="718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9" Oct 30 18:48:56.026113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.026090 6966 generic.go:356] "Generic (PLEG): container finished" podID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerID="0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4" exitCode=0 Oct 30 18:48:56.026228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.026171 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:48:56.026228 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.026192 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerDied","Data":"0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4"} Oct 30 18:48:56.026337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.026243 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"008637e6-bef8-4abf-8849-b5cf9a3cb6d0","Type":"ContainerDied","Data":"75a5c34b578e298c3ecf785d3df9e8d7bec26b6752755ed2a021a70371432a81"} Oct 30 18:48:56.041432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.041411 6966 scope.go:122] "RemoveContainer" containerID="718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9" Oct 30 18:48:56.042034 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:56.041818 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9\": container with ID starting with 718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9 not found: ID does not exist" containerID="718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9" Oct 30 18:48:56.042034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.041854 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9"} err="failed to get container status \"718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9\": rpc error: code = NotFound desc = could not find container \"718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9\": container with ID starting with 718bb26708d661efe854a41fbd3f6003bde112403caf1ff12da43fb71fc1b6c9 not found: ID does not exist" Oct 30 18:48:56.042034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.041877 6966 scope.go:122] "RemoveContainer" containerID="3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5" Oct 30 18:48:56.051919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.051900 6966 scope.go:122] "RemoveContainer" containerID="0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4" Oct 30 18:48:56.078009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.077866 6966 scope.go:122] "RemoveContainer" containerID="4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c" Oct 30 18:48:56.078818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.078795 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:48:56.081996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.081917 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:56.083398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.083373 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:48:56.088380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.088301 6966 scope.go:122] "RemoveContainer" containerID="3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5" Oct 30 18:48:56.088636 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:56.088612 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5\": container with ID starting with 3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5 not found: ID does not exist" containerID="3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5" Oct 30 18:48:56.088712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.088645 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5"} err="failed to get container status \"3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5\": rpc error: code = NotFound desc = could not find container \"3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5\": container with ID starting with 3b4ff1836c7091cbfc24c476f201c39e462165e04e2abda63ee9f8acf91b2bd5 not found: ID does not exist" Oct 30 18:48:56.088712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.088671 6966 scope.go:122] "RemoveContainer" containerID="0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4" Oct 30 18:48:56.089013 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:56.088989 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4\": container with ID starting with 0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4 not found: ID does not exist" containerID="0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4" Oct 30 18:48:56.089081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.089021 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4"} err="failed to get container status \"0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4\": rpc error: code = NotFound desc = could not find container \"0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4\": container with ID starting with 0d534321c3a8d8e98f1391aab72aec2ce15168e9a3cad732d5994684b1679da4 not found: ID does not exist" Oct 30 18:48:56.089081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.089042 6966 scope.go:122] "RemoveContainer" containerID="4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c" Oct 30 18:48:56.089401 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:48:56.089382 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c\": container with ID starting with 4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c not found: ID does not exist" containerID="4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c" Oct 30 18:48:56.089517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.089409 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c"} err="failed to get container status \"4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c\": rpc error: code = NotFound desc = could not find container \"4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c\": container with ID starting with 4873f3164c8724e714ad3217af21d80e80d3a3687b46e106a9c04b65c764a26c not found: ID does not exist" Oct 30 18:48:56.117070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.117046 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:48:56.125421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.124889 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:48:56.314882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:56.314839 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:48:57.410588 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:57.410555 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" path="/var/lib/kubelet/pods/008637e6-bef8-4abf-8849-b5cf9a3cb6d0/volumes" Oct 30 18:48:57.411231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:57.411217 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f06aa0-5c25-40a9-b18d-f58ab7566349" path="/var/lib/kubelet/pods/73f06aa0-5c25-40a9-b18d-f58ab7566349/volumes" Oct 30 18:48:57.766453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:48:57.766370 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.606535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.606493 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.739089 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739058 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm"] Oct 30 18:49:16.739629 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739605 6966 cpu_manager.go:414] "containerMap: removing container" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="init-config-reloader" Oct 30 18:49:16.739629 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739629 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="init-config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739643 6966 cpu_manager.go:414] "containerMap: removing container" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="thanos-sidecar" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739650 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="thanos-sidecar" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739679 6966 cpu_manager.go:414] "containerMap: removing container" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739686 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739699 6966 cpu_manager.go:414] "containerMap: removing container" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="prometheus" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739707 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="prometheus" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739721 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="method-gate" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739729 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="method-gate" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739738 6966 cpu_manager.go:414] "containerMap: removing container" podUID="73f06aa0-5c25-40a9-b18d-f58ab7566349" containerName="otc-container" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739744 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f06aa0-5c25-40a9-b18d-f58ab7566349" containerName="otc-container" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739755 6966 cpu_manager.go:414] "containerMap: removing container" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="init-config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739761 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="init-config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739777 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fe872b97-2748-4e6c-9354-1307e4cfdc67" containerName="korrel8r" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739783 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe872b97-2748-4e6c-9354-1307e4cfdc67" containerName="korrel8r" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739795 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="prom-label-proxy" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739802 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="prom-label-proxy" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739810 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="kube-rbac-proxy" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739817 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="kube-rbac-proxy" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739836 6966 cpu_manager.go:414] "containerMap: removing container" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739843 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739860 6966 cpu_manager.go:414] "containerMap: removing container" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="alertmanager" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739867 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="alertmanager" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739938 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739946 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="008637e6-bef8-4abf-8849-b5cf9a3cb6d0" containerName="alertmanager" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739953 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="thanos-sidecar" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739963 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="config-reloader" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739973 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="05d1fed0-0d94-4859-b4bb-7e53696e21f4" containerName="prometheus" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739983 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="method-gate" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.739991 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f06aa0-5c25-40a9-b18d-f58ab7566349" containerName="otc-container" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.740000 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe872b97-2748-4e6c-9354-1307e4cfdc67" containerName="korrel8r" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.740006 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="prom-label-proxy" Oct 30 18:49:16.740394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.740014 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c55512d4-feb2-4d0f-884e-3ceddffda861" containerName="kube-rbac-proxy" Oct 30 18:49:16.744888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.744867 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.754472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.754430 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.754581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.754545 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.754644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.754581 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.754694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.754679 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f7gt\" (UniqueName: \"kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.754739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.754703 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.754739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.754733 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.759552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.759530 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:49:16.759654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.759602 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-ca-bundle\"" Oct 30 18:49:16.759706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.759664 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:49:16.770648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.770626 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-usage-logs-loki-secret\"" Oct 30 18:49:16.770648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.770629 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-config\"" Oct 30 18:49:16.770784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.770664 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-grpc\"" Oct 30 18:49:16.770973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.770958 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-6pb2l\"" Oct 30 18:49:16.771071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.770977 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-querier-http\"" Oct 30 18:49:16.772313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.772293 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm"] Oct 30 18:49:16.851030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.850995 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:16.855208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.855180 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.855361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.855220 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.855361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.855245 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.855361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.855274 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f7gt\" (UniqueName: \"kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.855361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.855292 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.855361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.855320 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.856214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.856190 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.856214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.856195 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.857964 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.857913 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.858258 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.858236 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.858963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.858941 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.880703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.880685 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f7gt\" (UniqueName: \"kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt\") pod \"data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:16.901050 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.901028 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc"] Oct 30 18:49:16.906487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.906453 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:16.925348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.925326 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc"] Oct 30 18:49:16.955822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.955795 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:16.955942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.955828 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:16.955942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.955920 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:16.956031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.955954 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:16.956031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.956007 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtnl\" (UniqueName: \"kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:16.974695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.974671 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-grpc\"" Oct 30 18:49:16.974796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:16.974753 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-http\"" Oct 30 18:49:17.056684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.056655 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.056848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.056692 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.056848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.056721 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtnl\" (UniqueName: \"kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.056848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.056770 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.056848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.056790 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.057407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.057381 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.057557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.057418 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:17.057557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.057512 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.059249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.059222 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.059350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.059237 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.079920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.079898 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtnl\" (UniqueName: \"kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.085844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.085820 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.195729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.195660 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b"] Oct 30 18:49:17.200954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.200928 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.201476 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:49:17.201435 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:49:17.218660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.218638 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:17.222236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.222214 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b"] Oct 30 18:49:17.224365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.224339 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 30 18:49:17.224586 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.224521 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:49:17.259284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259253 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259305 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259346 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sq2h7\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259377 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259393 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259416 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259439 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.259840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.259485 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.291341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.291308 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 30 18:49:17.291484 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.291316 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:49:17.291552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.291487 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 30 18:49:17.291596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.291544 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-9wphs\"" Oct 30 18:49:17.345667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.345639 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.354761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.354741 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm"] Oct 30 18:49:17.359821 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:49:17.359763 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd08f6449_219f_48a5_bae4_3453c806d4df.slice/crio-7662b1b4f88cdd55385fb4a70c551244cf3fd66d40d65e975bba979f9c05c3de WatchSource:0}: Error finding container 7662b1b4f88cdd55385fb4a70c551244cf3fd66d40d65e975bba979f9c05c3de: Status 404 returned error can't find the container with id 7662b1b4f88cdd55385fb4a70c551244cf3fd66d40d65e975bba979f9c05c3de Oct 30 18:49:17.359929 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.359904 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.359937 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.359983 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360027 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360114 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360059 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360111 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360158 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360228 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sq2h7\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360703 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.360874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.360846 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.361097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.361072 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.362235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.362219 6966 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:49:17.362764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.362744 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.362968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.362938 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.362968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.362944 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.363101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.362999 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.381990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.381970 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sq2h7\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7\") pod \"data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.516331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.516299 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc"] Oct 30 18:49:17.521485 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:49:17.521444 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf1b6f47_2b39_4531_b7ce_88bf1b301a79.slice/crio-43416850c6f42e0cb96aafc57bd8f8f08f23cbd6efaf4576944005d80a355760 WatchSource:0}: Error finding container 43416850c6f42e0cb96aafc57bd8f8f08f23cbd6efaf4576944005d80a355760: Status 404 returned error can't find the container with id 43416850c6f42e0cb96aafc57bd8f8f08f23cbd6efaf4576944005d80a355760 Oct 30 18:49:17.530011 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.529992 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-9wphs\"" Oct 30 18:49:17.537909 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.537881 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:17.569697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.569670 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:17.861546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:17.856591 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b"] Oct 30 18:49:17.866285 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:49:17.866241 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09ed9570_561f_4c0c_828d_ca47342e2fd4.slice/crio-c767ee23fd9f8382639fa815f05ee4eeddecae14942a779dba779fc40ec87a81 WatchSource:0}: Error finding container c767ee23fd9f8382639fa815f05ee4eeddecae14942a779dba779fc40ec87a81: Status 404 returned error can't find the container with id c767ee23fd9f8382639fa815f05ee4eeddecae14942a779dba779fc40ec87a81 Oct 30 18:49:18.141440 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:18.141359 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" event={"ID":"d08f6449-219f-48a5-bae4-3453c806d4df","Type":"ContainerStarted","Data":"7662b1b4f88cdd55385fb4a70c551244cf3fd66d40d65e975bba979f9c05c3de"} Oct 30 18:49:18.142580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:18.142555 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" event={"ID":"09ed9570-561f-4c0c-828d-ca47342e2fd4","Type":"ContainerStarted","Data":"c767ee23fd9f8382639fa815f05ee4eeddecae14942a779dba779fc40ec87a81"} Oct 30 18:49:18.143789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:18.143768 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" event={"ID":"df1b6f47-2b39-4531-b7ce-88bf1b301a79","Type":"ContainerStarted","Data":"43416850c6f42e0cb96aafc57bd8f8f08f23cbd6efaf4576944005d80a355760"} Oct 30 18:49:19.191209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:19.191173 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:19.417037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:19.416999 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:19.886833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:19.886718 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:20.112357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:20.112318 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:20.334712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:20.334605 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:21.313772 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:21.313731 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:22.172587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.172494 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" event={"ID":"09ed9570-561f-4c0c-828d-ca47342e2fd4","Type":"ContainerStarted","Data":"53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1"} Oct 30 18:49:22.174168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.174137 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" event={"ID":"df1b6f47-2b39-4531-b7ce-88bf1b301a79","Type":"ContainerStarted","Data":"b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8"} Oct 30 18:49:22.174319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.174184 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:22.175619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.175598 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" event={"ID":"d08f6449-219f-48a5-bae4-3453c806d4df","Type":"ContainerStarted","Data":"6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668"} Oct 30 18:49:22.175731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.175720 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:22.214323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.214280 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" podStartSLOduration=1.8268914170000001 podStartE2EDuration="6.21426837s" totalImagesPullingTime="4.387376953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:22.213933753 +0000 UTC m=+3124.920119111" watchObservedRunningTime="2026-10-30 18:49:22.21426837 +0000 UTC m=+3124.920453726" Oct 30 18:49:22.256301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.256260 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" podStartSLOduration=1.7142678139999998 podStartE2EDuration="6.256249052s" totalImagesPullingTime="4.541981238s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:22.254547855 +0000 UTC m=+3124.960733214" watchObservedRunningTime="2026-10-30 18:49:22.256249052 +0000 UTC m=+3124.962434408" Oct 30 18:49:22.923907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:22.923871 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:23.182795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:23.182736 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" event={"ID":"09ed9570-561f-4c0c-828d-ca47342e2fd4","Type":"ContainerStarted","Data":"65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129"} Oct 30 18:49:23.241996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:23.241952 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" podStartSLOduration=1.090410415 podStartE2EDuration="6.241939453s" totalImagesPullingTime="5.151529038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:49:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:49:23.229581779 +0000 UTC m=+3125.935767138" watchObservedRunningTime="2026-10-30 18:49:23.241939453 +0000 UTC m=+3125.948124810" Oct 30 18:49:23.916359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:23.916325 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:24.188084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:24.187998 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:24.188084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:24.188043 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:24.194698 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:24.194674 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:24.197125 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:24.197105 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:49:41.195779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:41.195744 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:41.413671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:41.413639 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:49:43.188716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:43.188687 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:49:43.189205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:49:43.189040 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:49:47.198320 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:49:47.198281 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:50:17.201071 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:50:17.200993 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:50:22.774513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:22.774483 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:23.032087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:23.031997 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:24.034717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:24.034682 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:39.998573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:39.998533 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:47.199025 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:50:47.198989 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:50:50.482400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:50.482365 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:51.950479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:51.950431 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr"] Oct 30 18:50:51.954177 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:51.954123 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:51.989246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:51.989220 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr"] Oct 30 18:50:52.030144 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.030114 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:52.044258 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.044236 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rl5v4\"" Oct 30 18:50:52.044410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.044392 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 18:50:52.044410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.044405 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 18:50:52.097875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.097852 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.097988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.097883 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.098036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.098015 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz4hd\" (UniqueName: \"kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.198891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.198864 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz4hd\" (UniqueName: \"kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.198986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.198923 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.199048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.199028 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.199649 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.199629 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.201287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.201242 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.221404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.221380 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz4hd\" (UniqueName: \"kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd\") pod \"data-science-usage-logs-collector-5bc884688f-hvpbr\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.251942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.251917 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:52.267512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.267495 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:52.562567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.562540 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr"] Oct 30 18:50:52.567241 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:50:52.567212 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce2c15da_64d1_4f18_9a99_5f25ebbc9e36.slice/crio-46a79e8e69325c64abcc73bbac18a5517ee2846f5387240366e3dd676d01ae41 WatchSource:0}: Error finding container 46a79e8e69325c64abcc73bbac18a5517ee2846f5387240366e3dd676d01ae41: Status 404 returned error can't find the container with id 46a79e8e69325c64abcc73bbac18a5517ee2846f5387240366e3dd676d01ae41 Oct 30 18:50:52.569521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.569488 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:50:52.569595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.569546 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:50:52.610628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:52.610606 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" event={"ID":"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36","Type":"ContainerStarted","Data":"46a79e8e69325c64abcc73bbac18a5517ee2846f5387240366e3dd676d01ae41"} Oct 30 18:50:53.616535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:53.616502 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" event={"ID":"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36","Type":"ContainerStarted","Data":"6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7"} Oct 30 18:50:53.657409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:53.657362 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" podStartSLOduration=2.657349831 podStartE2EDuration="2.657349831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:53.656299475 +0000 UTC m=+3216.362484834" watchObservedRunningTime="2026-10-30 18:50:53.657349831 +0000 UTC m=+3216.363535188" Oct 30 18:50:55.524281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.524244 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fq5zj"] Oct 30 18:50:55.528333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.528310 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.553277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.553246 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:50:55.553397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.553281 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:50:55.553397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.553374 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-czlzw\"" Oct 30 18:50:55.554999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.554979 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fq5zj"] Oct 30 18:50:55.622966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.622933 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:55.626320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.626294 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.626440 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.626322 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.626440 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.626363 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25td4\" (UniqueName: \"kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.727224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.727200 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.727351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.727229 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.727351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.727278 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25td4\" (UniqueName: \"kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.728004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.727982 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.729715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.729695 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.749553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.749530 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25td4\" (UniqueName: \"kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4\") pod \"korrel8r-5f5775896-fq5zj\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.841961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.841940 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:55.852136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:55.852109 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:56.140728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.140654 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fq5zj"] Oct 30 18:50:56.145945 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:50:56.145919 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcb1827_cc31_4c3c_a68e_d69ddc40db03.slice/crio-94f32def4109c001941d009dd42ea8c5e2eaea6c129ca97effc095ddd72937d3 WatchSource:0}: Error finding container 94f32def4109c001941d009dd42ea8c5e2eaea6c129ca97effc095ddd72937d3: Status 404 returned error can't find the container with id 94f32def4109c001941d009dd42ea8c5e2eaea6c129ca97effc095ddd72937d3 Oct 30 18:50:56.441513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.441406 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:56.633174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.633139 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" event={"ID":"0bcb1827-cc31-4c3c-a68e-d69ddc40db03","Type":"ContainerStarted","Data":"8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce"} Oct 30 18:50:56.633572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.633190 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" event={"ID":"0bcb1827-cc31-4c3c-a68e-d69ddc40db03","Type":"ContainerStarted","Data":"94f32def4109c001941d009dd42ea8c5e2eaea6c129ca97effc095ddd72937d3"} Oct 30 18:50:56.633572 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.633248 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:50:56.676289 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.676245 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" podStartSLOduration=1.676232468 podStartE2EDuration="1.676232468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:56.673587898 +0000 UTC m=+3219.379773252" watchObservedRunningTime="2026-10-30 18:50:56.676232468 +0000 UTC m=+3219.382417825" Oct 30 18:50:56.743027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:56.742966 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:57.153098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.153063 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr"] Oct 30 18:50:57.153286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.153238 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" podUID="ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" containerName="otc-container" containerID="cri-o://6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7" gracePeriod=30 Oct 30 18:50:57.408115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.408061 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:57.542182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.542135 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal\") pod \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " Oct 30 18:50:57.542349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.542230 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs\") pod \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " Oct 30 18:50:57.542349 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.542281 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd\" (UniqueName: \"kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd\") pod \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\" (UID: \"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36\") " Oct 30 18:50:57.542518 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.542486 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal" pod "ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" (UID: "ce2c15da-64d1-4f18-9a99-5f25ebbc9e36"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:57.542663 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.542617 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:50:57.544383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.544362 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd" pod "ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" (UID: "ce2c15da-64d1-4f18-9a99-5f25ebbc9e36"). InnerVolumeSpecName "kube-api-access-qz4hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:57.544487 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.544392 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs" pod "ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" (UID: "ce2c15da-64d1-4f18-9a99-5f25ebbc9e36"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:57.640877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.640848 6966 generic.go:356] "Generic (PLEG): container finished" podID="ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" containerID="6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7" exitCode=0 Oct 30 18:50:57.641280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.640907 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" Oct 30 18:50:57.641280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.640936 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" event={"ID":"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36","Type":"ContainerDied","Data":"6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7"} Oct 30 18:50:57.641280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.640988 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr" event={"ID":"ce2c15da-64d1-4f18-9a99-5f25ebbc9e36","Type":"ContainerDied","Data":"46a79e8e69325c64abcc73bbac18a5517ee2846f5387240366e3dd676d01ae41"} Oct 30 18:50:57.641280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.641020 6966 scope.go:122] "RemoveContainer" containerID="6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7" Oct 30 18:50:57.643473 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.643435 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz4hd\" (UniqueName: \"kubernetes.io/projected/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-kube-api-access-qz4hd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:50:57.643609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.643482 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:50:57.650493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.650444 6966 scope.go:122] "RemoveContainer" containerID="6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7" Oct 30 18:50:57.650775 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:50:57.650757 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7\": container with ID starting with 6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7 not found: ID does not exist" containerID="6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7" Oct 30 18:50:57.650836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.650783 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7"} err="failed to get container status \"6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7\": rpc error: code = NotFound desc = could not find container \"6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7\": container with ID starting with 6ef9ab2af52bcf4f892faaeda56612cc1a9bb6fdfd1504e8146fc4f7fd4271f7 not found: ID does not exist" Oct 30 18:50:57.693304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.693245 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr"] Oct 30 18:50:57.697320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.697299 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-5bc884688f-hvpbr"] Oct 30 18:50:57.778449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.778416 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:57.946015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.945942 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45"] Oct 30 18:50:57.946298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.946277 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" containerName="otc-container" Oct 30 18:50:57.946298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.946293 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" containerName="otc-container" Oct 30 18:50:57.946503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.946409 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" containerName="otc-container" Oct 30 18:50:57.951077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.951050 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:57.965946 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.965925 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-ca-bundle\"" Oct 30 18:50:57.966261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.966224 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-dockercfg-h78x9\"" Oct 30 18:50:57.966401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.966303 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-http\"" Oct 30 18:50:57.966401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.966373 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-query-frontend-grpc\"" Oct 30 18:50:57.966544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.966439 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-config\"" Oct 30 18:50:57.971232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:57.971213 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45"] Oct 30 18:50:58.006715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.006687 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.046526 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.046499 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.046645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.046534 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8brz8\" (UniqueName: \"kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.046645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.046564 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.046750 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.046677 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.046750 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.046709 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.147545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.147516 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.147725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.147551 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8brz8\" (UniqueName: \"kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.147725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.147588 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.147725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.147650 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.147725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.147691 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.148287 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.148261 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.148412 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.148301 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.149993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.149966 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.150116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.150091 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.174648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.174623 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8brz8\" (UniqueName: \"kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8\") pod \"data-science-lokistack-query-frontend-5cddddc95c-v5c45\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.242006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.241936 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf"] Oct 30 18:50:58.245653 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.245619 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.245813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.245794 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.266209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.266190 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.270710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.270688 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 18:50:58.270986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.270877 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-ca-bundle\"" Oct 30 18:50:58.273347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.273327 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf"] Oct 30 18:50:58.325067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.324709 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-client-http\"" Oct 30 18:50:58.325067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.324804 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway\"" Oct 30 18:50:58.325067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.324931 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-http\"" Oct 30 18:50:58.325067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.325019 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-gateway-dockercfg-8wvgb\"" Oct 30 18:50:58.349206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349182 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349214 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349243 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349359 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349387 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349407 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpvsw\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349432 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.349746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.349550 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450786 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450849 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450872 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450904 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450930 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450947 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.450972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450964 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpvsw\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.451379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.450986 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.451746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.451715 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.451746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.451736 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.451920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.451748 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.453621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.453601 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.453718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.453653 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.453978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.453958 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.454029 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.453984 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.472844 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.472824 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpvsw\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw\") pod \"data-science-lokistack-gateway-6f9b9f4d87-svslf\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.506151 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.506092 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.560754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.560721 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:58.568149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.568129 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45"] Oct 30 18:50:58.573780 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:50:58.573758 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c2fbe6c_cedf_4f1e_a19d_fdb918247882.slice/crio-326d325fe4056ba82ec634aecc05fc7de842ce7718bb9e5dd599271a3050200c WatchSource:0}: Error finding container 326d325fe4056ba82ec634aecc05fc7de842ce7718bb9e5dd599271a3050200c: Status 404 returned error can't find the container with id 326d325fe4056ba82ec634aecc05fc7de842ce7718bb9e5dd599271a3050200c Oct 30 18:50:58.650340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.650307 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" event={"ID":"6c2fbe6c-cedf-4f1e-a19d-fdb918247882","Type":"ContainerStarted","Data":"567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3"} Oct 30 18:50:58.650340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.650353 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" event={"ID":"6c2fbe6c-cedf-4f1e-a19d-fdb918247882","Type":"ContainerStarted","Data":"326d325fe4056ba82ec634aecc05fc7de842ce7718bb9e5dd599271a3050200c"} Oct 30 18:50:58.651157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.650416 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:50:58.692455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.692397 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" podStartSLOduration=1.6923819039999999 podStartE2EDuration="1.692381904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:58.690148108 +0000 UTC m=+3221.396333465" watchObservedRunningTime="2026-10-30 18:50:58.692381904 +0000 UTC m=+3221.398567261" Oct 30 18:50:58.730048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.730014 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:50:58.868697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:58.868631 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf"] Oct 30 18:50:58.873072 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:50:58.873033 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod74cb7faa_cb74_4097_aa10_9d4b9c474a10.slice/crio-ffd76fee5e6db08273b9e72fce7c41e8a98b91d5b654eaace6584ceb5cc81fc4 WatchSource:0}: Error finding container ffd76fee5e6db08273b9e72fce7c41e8a98b91d5b654eaace6584ceb5cc81fc4: Status 404 returned error can't find the container with id ffd76fee5e6db08273b9e72fce7c41e8a98b91d5b654eaace6584ceb5cc81fc4 Oct 30 18:50:59.412607 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.412577 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce2c15da-64d1-4f18-9a99-5f25ebbc9e36" path="/var/lib/kubelet/pods/ce2c15da-64d1-4f18-9a99-5f25ebbc9e36/volumes" Oct 30 18:50:59.657652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.657611 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" event={"ID":"74cb7faa-cb74-4097-aa10-9d4b9c474a10","Type":"ContainerStarted","Data":"ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e"} Oct 30 18:50:59.657652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.657658 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" event={"ID":"74cb7faa-cb74-4097-aa10-9d4b9c474a10","Type":"ContainerStarted","Data":"15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861"} Oct 30 18:50:59.658138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.657774 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:59.658138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.658036 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:59.658138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.658060 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" event={"ID":"74cb7faa-cb74-4097-aa10-9d4b9c474a10","Type":"ContainerStarted","Data":"ffd76fee5e6db08273b9e72fce7c41e8a98b91d5b654eaace6584ceb5cc81fc4"} Oct 30 18:50:59.663750 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.663690 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:59.666493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.666442 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:50:59.701439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:50:59.701395 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" podStartSLOduration=1.701382806 podStartE2EDuration="1.701382806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:50:59.700743481 +0000 UTC m=+3222.406928837" watchObservedRunningTime="2026-10-30 18:50:59.701382806 +0000 UTC m=+3222.407568162" Oct 30 18:51:00.222648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.222608 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:00.445533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.445504 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:00.899376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.899337 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 18:51:00.903547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.903523 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:00.917691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.917673 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-logs-loki-secret\"" Oct 30 18:51:00.928775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.928756 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-http\"" Oct 30 18:51:00.929542 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.929521 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-lokistack-compactor-grpc\"" Oct 30 18:51:00.931880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.931858 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 18:51:00.973220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.973197 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:00.973320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.973235 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:00.973320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.973257 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7225\" (UniqueName: \"kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:00.973404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.973377 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:00.973449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.973417 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:00.973449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:00.973441 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.010477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.010433 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.073972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.073951 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.073984 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.074013 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.074045 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.074066 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7225\" (UniqueName: \"kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.074108 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074845 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.074821 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.074944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.074849 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.076519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.076476 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.076771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.076748 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.076845 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.076829 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.096413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.096389 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7225\" (UniqueName: \"kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:01.229664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.229599 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:01.817357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.817330 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bwwdm"] Oct 30 18:51:01.822329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.822305 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.845178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.845143 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bwwdm"] Oct 30 18:51:01.881110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881082 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881112 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881144 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881239 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881278 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881346 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881378 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881431 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881475 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881497 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwt4t\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.881576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.881520 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.906578 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.906558 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 30 18:51:01.906937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.906705 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 30 18:51:01.906937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.906738 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-hnvcg\"" Oct 30 18:51:01.906937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.906812 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 30 18:51:01.929876 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.929854 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 30 18:51:01.982063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982033 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982068 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982116 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982141 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982204 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982178 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982238 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982282 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982308 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982346 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982372 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwt4t\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982408 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:51:01.982517 6966 secret.go:189] Couldn't get secret openshift-logging/data-science-cluster-log-forwarder-metrics: secret "data-science-cluster-log-forwarder-metrics" not found Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982551 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:51:01.982580 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics podName:d0c5a420-0122-4f24-89f6-a1f41953c51f nodeName:}" failed. No retries permitted until 2026-10-30 18:51:02.482558739 +0000 UTC m=+3225.188744089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics" (UniqueName: "kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics") pod "data-science-cluster-log-forwarder-bwwdm" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f") : secret "data-science-cluster-log-forwarder-metrics" not found Oct 30 18:51:01.982587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982580 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.983160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982950 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.983160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.982999 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.983275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.983214 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.983275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.983250 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.984345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.984326 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:01.984729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:01.984712 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.006132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.006110 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.006252 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.006235 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwt4t\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.107964 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.107910 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:02.207380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.207348 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bwwdm"] Oct 30 18:51:02.207627 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:51:02.207602 6966 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" podUID="d0c5a420-0122-4f24-89f6-a1f41953c51f" Oct 30 18:51:02.336557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.336520 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:02.487532 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.487447 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.489805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.489787 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\") pod \"data-science-cluster-log-forwarder-bwwdm\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.672991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.672955 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.681392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.681367 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:02.790149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790090 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790133 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790161 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790190 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790360 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790394 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790440 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790498 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790524 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:02.790738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790535 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790561 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790593 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods\") pod \"d0c5a420-0122-4f24-89f6-a1f41953c51f\" (UID: \"d0c5a420-0122-4f24-89f6-a1f41953c51f\") " Oct 30 18:51:02.790943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790827 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:02.790943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790871 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:51:02.790943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790890 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:02.790943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790896 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:51:02.791169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790964 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:51:02.791169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790974 6966 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-entrypoint\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.791169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.790997 6966 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config-openshift-service-cacrt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.791169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.791015 6966 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-datadir\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.791169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.791030 6966 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/d0c5a420-0122-4f24-89f6-a1f41953c51f-varlogpods\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.791169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.791044 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.792543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.792507 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "kube-api-access-mwt4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:02.792694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.792674 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:51:02.792883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.792860 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:51:02.792950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.792892 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:51:02.793056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.793042 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp" pod "d0c5a420-0122-4f24-89f6-a1f41953c51f" (UID: "d0c5a420-0122-4f24-89f6-a1f41953c51f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:51:02.892358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.892337 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.892358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.892357 6966 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-sa-token\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.892507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.892367 6966 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/d0c5a420-0122-4f24-89f6-a1f41953c51f-metrics\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.892507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.892375 6966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d0c5a420-0122-4f24-89f6-a1f41953c51f-trusted-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.892507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.892383 6966 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d0c5a420-0122-4f24-89f6-a1f41953c51f-tmp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:02.892507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:02.892390 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwt4t\" (UniqueName: \"kubernetes.io/projected/d0c5a420-0122-4f24-89f6-a1f41953c51f-kube-api-access-mwt4t\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:51:03.677950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.677926 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-bwwdm" Oct 30 18:51:03.753898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.753868 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bwwdm"] Oct 30 18:51:03.760102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.760078 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-bwwdm"] Oct 30 18:51:03.775625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.775600 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-zgmkr"] Oct 30 18:51:03.780434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.780417 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.794520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.794498 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-metrics\"" Oct 30 18:51:03.794659 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.794639 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-dockercfg-hnvcg\"" Oct 30 18:51:03.794736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.794703 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-config\"" Oct 30 18:51:03.805752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.805731 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-collector-token\"" Oct 30 18:51:03.806590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.806569 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-logging\"/\"data-science-cluster-log-forwarder-trustbundle\"" Oct 30 18:51:03.808158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.808133 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-zgmkr"] Oct 30 18:51:03.899660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899631 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899680 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899770 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899809 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899791 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899810 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899826 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899883 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtf69\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.899953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.899915 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.900093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.900021 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.900093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.900045 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:03.900093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:03.900061 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000517 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000551 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000572 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000598 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000620 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtf69\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000648 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000737 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000769 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000791 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.000868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000838 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.001215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000896 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.001215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.000948 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.001215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.001000 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.001426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.001398 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.001543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.001430 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.001964 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.001938 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.002073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.001948 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.003359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.003338 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.003502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.003361 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.004756 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.004736 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.023010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.022991 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.023491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.023445 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtf69\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69\") pod \"data-science-cluster-log-forwarder-zgmkr\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.092890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.092870 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:51:04.408173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.408140 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-zgmkr"] Oct 30 18:51:04.412886 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:51:04.412860 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24b54b4d_390a_4331_9b0a_551834333310.slice/crio-653b3807f6cf3afcb4c5937db83295f7b47c7ccb7cee6fd13ebfe0c71da906f9 WatchSource:0}: Error finding container 653b3807f6cf3afcb4c5937db83295f7b47c7ccb7cee6fd13ebfe0c71da906f9: Status 404 returned error can't find the container with id 653b3807f6cf3afcb4c5937db83295f7b47c7ccb7cee6fd13ebfe0c71da906f9 Oct 30 18:51:04.606375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.606341 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:04.633622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.633594 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") device path: \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:04.683865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.683806 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" event={"ID":"24b54b4d-390a-4331-9b0a-551834333310","Type":"ContainerStarted","Data":"653b3807f6cf3afcb4c5937db83295f7b47c7ccb7cee6fd13ebfe0c71da906f9"} Oct 30 18:51:04.707229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.707195 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:04.707334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.707276 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:04.720913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.720893 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") DevicePath \"csi-6112057b00a5460799d60e94c2b0618d4339163956b487ae0109084bbd0cec7d\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:04.943664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.943605 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/6ad14cb15e9ea1cfcd0eada63872bd059f13af41931d537dc1825fa19f87222c/globalmount\"" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:04.967108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:04.967083 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"data-science-lokistack-compactor-0\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:05.115812 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.115787 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:05.411705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.411669 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0c5a420-0122-4f24-89f6-a1f41953c51f" path="/var/lib/kubelet/pods/d0c5a420-0122-4f24-89f6-a1f41953c51f/volumes" Oct 30 18:51:05.420259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.420218 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 18:51:05.424817 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:51:05.424792 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3b1af291_85bb_4df5_9b8b_9c38baa54023.slice/crio-45d61954326f6f4738da52d5904dadad3ac7c3a2e69226eff9fa6a864912d5ec WatchSource:0}: Error finding container 45d61954326f6f4738da52d5904dadad3ac7c3a2e69226eff9fa6a864912d5ec: Status 404 returned error can't find the container with id 45d61954326f6f4738da52d5904dadad3ac7c3a2e69226eff9fa6a864912d5ec Oct 30 18:51:05.690602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.690516 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"3b1af291-85bb-4df5-9b8b-9c38baa54023","Type":"ContainerStarted","Data":"b1336791803486eaa3d921dfdf0c7fe8583f7f06139366a267f446e3eebb7a77"} Oct 30 18:51:05.690602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.690556 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"3b1af291-85bb-4df5-9b8b-9c38baa54023","Type":"ContainerStarted","Data":"45d61954326f6f4738da52d5904dadad3ac7c3a2e69226eff9fa6a864912d5ec"} Oct 30 18:51:05.690602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.690581 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:05.733949 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:05.733905 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podStartSLOduration=8.733892902000001 podStartE2EDuration="8.733892902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:05.732805218 +0000 UTC m=+3228.438990576" watchObservedRunningTime="2026-10-30 18:51:05.733892902 +0000 UTC m=+3228.440078258" Oct 30 18:51:07.643275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:07.643244 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:51:08.710613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:08.710575 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" event={"ID":"24b54b4d-390a-4331-9b0a-551834333310","Type":"ContainerStarted","Data":"b5f7b2c3b4c40ce0d51b4a730bdbe565ef338e72daa0aeab38b121dab27e2f24"} Oct 30 18:51:08.777124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:08.776939 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" podStartSLOduration=1.688660273 podStartE2EDuration="5.776924273s" totalImagesPullingTime="4.088264s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:51:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:51:08.765866766 +0000 UTC m=+3231.472052120" watchObservedRunningTime="2026-10-30 18:51:08.776924273 +0000 UTC m=+3231.483109633" Oct 30 18:51:17.199913 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:51:17.199872 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:51:19.664034 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:19.664006 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:51:23.165564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:23.165520 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:24.172383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:24.172347 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:51:26.702591 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:51:26.702562 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:51:47.199948 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:51:47.199912 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:52:07.989723 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:07.989685 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:08.236560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:08.236523 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:14.764009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:14.763974 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:17.200231 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:17.200192 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:52:21.483296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.483261 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fq5zj"] Oct 30 18:52:21.483752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.483515 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" podUID="0bcb1827-cc31-4c3c-a68e-d69ddc40db03" containerName="korrel8r" containerID="cri-o://8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce" gracePeriod=30 Oct 30 18:52:21.650116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.649995 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:21.720537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.720496 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 18:52:21.721098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.720756 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" podUID="3b1af291-85bb-4df5-9b8b-9c38baa54023" containerName="loki-compactor" containerID="cri-o://b1336791803486eaa3d921dfdf0c7fe8583f7f06139366a267f446e3eebb7a77" gracePeriod=30 Oct 30 18:52:21.736656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.736156 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45"] Oct 30 18:52:21.737282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.737241 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" podUID="6c2fbe6c-cedf-4f1e-a19d-fdb918247882" containerName="loki-query-frontend" containerID="cri-o://567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3" gracePeriod=30 Oct 30 18:52:21.740354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.740319 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf"] Oct 30 18:52:21.741534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.740490 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="gateway" containerID="cri-o://15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861" gracePeriod=30 Oct 30 18:52:21.741534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.741428 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="opa" containerID="cri-o://ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e" gracePeriod=30 Oct 30 18:52:21.828775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.828740 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-zgmkr"] Oct 30 18:52:21.829027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.828969 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" podUID="24b54b4d-390a-4331-9b0a-551834333310" containerName="collector" containerID="cri-o://b5f7b2c3b4c40ce0d51b4a730bdbe565ef338e72daa0aeab38b121dab27e2f24" gracePeriod=10 Oct 30 18:52:21.914291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.914259 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:21.942125 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.942099 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:52:21.968062 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.968032 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls\" (UniqueName: \"kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls\") pod \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " Oct 30 18:52:21.968198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.968141 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config\" (UniqueName: \"kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config\") pod \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " Oct 30 18:52:21.968270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.968217 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4\" (UniqueName: \"kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4\") pod \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\" (UID: \"0bcb1827-cc31-4c3c-a68e-d69ddc40db03\") " Oct 30 18:52:21.968530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.968504 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config" pod "0bcb1827-cc31-4c3c-a68e-d69ddc40db03" (UID: "0bcb1827-cc31-4c3c-a68e-d69ddc40db03"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:21.968708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.968685 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:21.970861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.970833 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls" pod "0bcb1827-cc31-4c3c-a68e-d69ddc40db03" (UID: "0bcb1827-cc31-4c3c-a68e-d69ddc40db03"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:21.971782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:21.971756 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4" pod "0bcb1827-cc31-4c3c-a68e-d69ddc40db03" (UID: "0bcb1827-cc31-4c3c-a68e-d69ddc40db03"). InnerVolumeSpecName "kube-api-access-25td4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.014236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.014207 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:52:22.069705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.069671 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle\") pod \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " Oct 30 18:52:22.069889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.069768 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8\" (UniqueName: \"kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8\") pod \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " Oct 30 18:52:22.069889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.069825 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config\") pod \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " Oct 30 18:52:22.070028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.069894 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http\") pod \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " Oct 30 18:52:22.070028 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.069939 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc\") pod \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\" (UID: \"6c2fbe6c-cedf-4f1e-a19d-fdb918247882\") " Oct 30 18:52:22.070141 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.070091 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle" pod "6c2fbe6c-cedf-4f1e-a19d-fdb918247882" (UID: "6c2fbe6c-cedf-4f1e-a19d-fdb918247882"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.070413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.070382 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.070533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.070416 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25td4\" (UniqueName: \"kubernetes.io/projected/0bcb1827-cc31-4c3c-a68e-d69ddc40db03-kube-api-access-25td4\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.070533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.070430 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.070533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.070444 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config" pod "6c2fbe6c-cedf-4f1e-a19d-fdb918247882" (UID: "6c2fbe6c-cedf-4f1e-a19d-fdb918247882"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.072247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.072214 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc" pod "6c2fbe6c-cedf-4f1e-a19d-fdb918247882" (UID: "6c2fbe6c-cedf-4f1e-a19d-fdb918247882"). InnerVolumeSpecName "data-science-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.072541 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.072515 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8" pod "6c2fbe6c-cedf-4f1e-a19d-fdb918247882" (UID: "6c2fbe6c-cedf-4f1e-a19d-fdb918247882"). InnerVolumeSpecName "kube-api-access-8brz8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.072851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.072823 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http" pod "6c2fbe6c-cedf-4f1e-a19d-fdb918247882" (UID: "6c2fbe6c-cedf-4f1e-a19d-fdb918247882"). InnerVolumeSpecName "data-science-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.080936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.080912 6966 generic.go:356] "Generic (PLEG): container finished" podID="0bcb1827-cc31-4c3c-a68e-d69ddc40db03" containerID="8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce" exitCode=0 Oct 30 18:52:22.081056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.080955 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" Oct 30 18:52:22.081056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.080999 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" event={"ID":"0bcb1827-cc31-4c3c-a68e-d69ddc40db03","Type":"ContainerDied","Data":"8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce"} Oct 30 18:52:22.081162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.081059 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5f5775896-fq5zj" event={"ID":"0bcb1827-cc31-4c3c-a68e-d69ddc40db03","Type":"ContainerDied","Data":"94f32def4109c001941d009dd42ea8c5e2eaea6c129ca97effc095ddd72937d3"} Oct 30 18:52:22.081162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.081090 6966 scope.go:122] "RemoveContainer" containerID="8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce" Oct 30 18:52:22.083360 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.083333 6966 generic.go:356] "Generic (PLEG): container finished" podID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerID="ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e" exitCode=0 Oct 30 18:52:22.083477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.083401 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" event={"ID":"74cb7faa-cb74-4097-aa10-9d4b9c474a10","Type":"ContainerDied","Data":"ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e"} Oct 30 18:52:22.085246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.085178 6966 generic.go:356] "Generic (PLEG): container finished" podID="24b54b4d-390a-4331-9b0a-551834333310" containerID="b5f7b2c3b4c40ce0d51b4a730bdbe565ef338e72daa0aeab38b121dab27e2f24" exitCode=0 Oct 30 18:52:22.085246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.085226 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" event={"ID":"24b54b4d-390a-4331-9b0a-551834333310","Type":"ContainerDied","Data":"b5f7b2c3b4c40ce0d51b4a730bdbe565ef338e72daa0aeab38b121dab27e2f24"} Oct 30 18:52:22.086693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.086666 6966 generic.go:356] "Generic (PLEG): container finished" podID="6c2fbe6c-cedf-4f1e-a19d-fdb918247882" containerID="567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3" exitCode=0 Oct 30 18:52:22.086787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.086692 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" event={"ID":"6c2fbe6c-cedf-4f1e-a19d-fdb918247882","Type":"ContainerDied","Data":"567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3"} Oct 30 18:52:22.086787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.086711 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" Oct 30 18:52:22.086787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.086721 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45" event={"ID":"6c2fbe6c-cedf-4f1e-a19d-fdb918247882","Type":"ContainerDied","Data":"326d325fe4056ba82ec634aecc05fc7de842ce7718bb9e5dd599271a3050200c"} Oct 30 18:52:22.091631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.091614 6966 scope.go:122] "RemoveContainer" containerID="8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce" Oct 30 18:52:22.091907 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:22.091883 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce\": container with ID starting with 8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce not found: ID does not exist" containerID="8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce" Oct 30 18:52:22.091975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.091916 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce"} err="failed to get container status \"8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce\": rpc error: code = NotFound desc = could not find container \"8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce\": container with ID starting with 8e35a4b7b1ec1830d1110b7a83d2d43004a872a4bfc26ab7db2c9de8b2e6cfce not found: ID does not exist" Oct 30 18:52:22.091975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.091931 6966 scope.go:122] "RemoveContainer" containerID="567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3" Oct 30 18:52:22.101843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.101749 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:52:22.102048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.101864 6966 scope.go:122] "RemoveContainer" containerID="567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3" Oct 30 18:52:22.102156 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:22.102134 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3\": container with ID starting with 567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3 not found: ID does not exist" containerID="567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3" Oct 30 18:52:22.102203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.102153 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3"} err="failed to get container status \"567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3\": rpc error: code = NotFound desc = could not find container \"567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3\": container with ID starting with 567386510a944e5a3dfe24f9a810e913e3eb640dc1e11f94ae2d4356dc4adcd3 not found: ID does not exist" Oct 30 18:52:22.131687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.131604 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fq5zj"] Oct 30 18:52:22.137268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.137231 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:22.138132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.138110 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5f5775896-fq5zj"] Oct 30 18:52:22.171007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.170984 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171032 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171053 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171247 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171290 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171382 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171341 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171382 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171375 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171338 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "varlogpods". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:52:22.171505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171379 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "config-openshift-service-cacrt". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.171505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171419 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171450 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171491 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171532 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir\") pod \"24b54b4d-390a-4331-9b0a-551834333310\" (UID: \"24b54b4d-390a-4331-9b0a-551834333310\") " Oct 30 18:52:22.171819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171736 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.171876 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171825 6966 reconciler_common.go:299] "Volume detached for volume \"varlogpods\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-varlogpods\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.171876 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171843 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.171876 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171859 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-data-science-lokistack-query-frontend-grpc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.172042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171875 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8brz8\" (UniqueName: \"kubernetes.io/projected/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-kube-api-access-8brz8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.172042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171890 6966 reconciler_common.go:299] "Volume detached for volume \"config-openshift-service-cacrt\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config-openshift-service-cacrt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.172042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171904 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6c2fbe6c-cedf-4f1e-a19d-fdb918247882-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.172042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171927 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.172042 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.171934 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "datadir". PluginName "kubernetes.io/host-path", VolumeGIDValue "" Oct 30 18:52:22.173873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.173744 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.173873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.173808 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "data-science-cluster-log-forwarder-collector-token". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.173873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.173853 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "metrics". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:22.174155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.174134 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "kube-api-access-qtf69". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:22.174249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.174228 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:52:22.176739 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.176719 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint" pod "24b54b4d-390a-4331-9b0a-551834333310" (UID: "24b54b4d-390a-4331-9b0a-551834333310"). InnerVolumeSpecName "entrypoint". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:22.226538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.226506 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45"] Oct 30 18:52:22.232414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.232387 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-query-frontend-5cddddc95c-v5c45"] Oct 30 18:52:22.273354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273303 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtf69\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-kube-api-access-qtf69\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273326 6966 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-trusted-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273337 6966 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/24b54b4d-390a-4331-9b0a-551834333310-tmp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273345 6966 reconciler_common.go:299] "Volume detached for volume \"entrypoint\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-entrypoint\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273353 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/24b54b4d-390a-4331-9b0a-551834333310-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273361 6966 reconciler_common.go:299] "Volume detached for volume \"metrics\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-metrics\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273368 6966 reconciler_common.go:299] "Volume detached for volume \"datadir\" (UniqueName: \"kubernetes.io/host-path/24b54b4d-390a-4331-9b0a-551834333310-datadir\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273375 6966 reconciler_common.go:299] "Volume detached for volume \"sa-token\" (UniqueName: \"kubernetes.io/projected/24b54b4d-390a-4331-9b0a-551834333310-sa-token\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.273648 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.273383 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-cluster-log-forwarder-collector-token\" (UniqueName: \"kubernetes.io/secret/24b54b4d-390a-4331-9b0a-551834333310-data-science-cluster-log-forwarder-collector-token\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:22.389855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:22.389828 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:23.006294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.006270 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:52:23.081352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081326 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081365 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081393 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081429 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081483 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081547 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081606 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.081859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.081685 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle\") pod \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\" (UID: \"74cb7faa-cb74-4097-aa10-9d4b9c474a10\") " Oct 30 18:52:23.082190 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.082008 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.084087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.084053 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.084181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.084159 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "data-science-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.084314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.084297 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "kube-api-access-tpvsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.086139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.086112 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "data-science-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.087108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.087085 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.088177 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.088155 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.089324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.089302 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants" pod "74cb7faa-cb74-4097-aa10-9d4b9c474a10" (UID: "74cb7faa-cb74-4097-aa10-9d4b9c474a10"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.093490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.093454 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" event={"ID":"24b54b4d-390a-4331-9b0a-551834333310","Type":"ContainerDied","Data":"653b3807f6cf3afcb4c5937db83295f7b47c7ccb7cee6fd13ebfe0c71da906f9"} Oct 30 18:52:23.093490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.093487 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-logging/data-science-cluster-log-forwarder-zgmkr" Oct 30 18:52:23.093658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.093534 6966 scope.go:122] "RemoveContainer" containerID="b5f7b2c3b4c40ce0d51b4a730bdbe565ef338e72daa0aeab38b121dab27e2f24" Oct 30 18:52:23.097038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.097015 6966 generic.go:356] "Generic (PLEG): container finished" podID="3b1af291-85bb-4df5-9b8b-9c38baa54023" containerID="b1336791803486eaa3d921dfdf0c7fe8583f7f06139366a267f446e3eebb7a77" exitCode=0 Oct 30 18:52:23.097126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.097088 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"3b1af291-85bb-4df5-9b8b-9c38baa54023","Type":"ContainerDied","Data":"b1336791803486eaa3d921dfdf0c7fe8583f7f06139366a267f446e3eebb7a77"} Oct 30 18:52:23.100547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.100520 6966 generic.go:356] "Generic (PLEG): container finished" podID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerID="15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861" exitCode=0 Oct 30 18:52:23.100660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.100575 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" Oct 30 18:52:23.100660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.100589 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" event={"ID":"74cb7faa-cb74-4097-aa10-9d4b9c474a10","Type":"ContainerDied","Data":"15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861"} Oct 30 18:52:23.100660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.100620 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf" event={"ID":"74cb7faa-cb74-4097-aa10-9d4b9c474a10","Type":"ContainerDied","Data":"ffd76fee5e6db08273b9e72fce7c41e8a98b91d5b654eaace6584ceb5cc81fc4"} Oct 30 18:52:23.106363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.106309 6966 scope.go:122] "RemoveContainer" containerID="ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e" Oct 30 18:52:23.115717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.115701 6966 scope.go:122] "RemoveContainer" containerID="15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861" Oct 30 18:52:23.124341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.124320 6966 scope.go:122] "RemoveContainer" containerID="ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e" Oct 30 18:52:23.124649 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:23.124621 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e\": container with ID starting with ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e not found: ID does not exist" containerID="ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e" Oct 30 18:52:23.124738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.124659 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e"} err="failed to get container status \"ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e\": rpc error: code = NotFound desc = could not find container \"ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e\": container with ID starting with ebe7435d8ac9ff617b4d11e9d94262f6c4f51454deb013114954ac6842353a5e not found: ID does not exist" Oct 30 18:52:23.124738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.124692 6966 scope.go:122] "RemoveContainer" containerID="15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861" Oct 30 18:52:23.124933 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:23.124917 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861\": container with ID starting with 15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861 not found: ID does not exist" containerID="15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861" Oct 30 18:52:23.125001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.124937 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861"} err="failed to get container status \"15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861\": rpc error: code = NotFound desc = could not find container \"15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861\": container with ID starting with 15eb3b70c1850c9f53226866907681c5c8a4113c41e77b53b5eefacf59981861 not found: ID does not exist" Oct 30 18:52:23.157296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.157266 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-zgmkr"] Oct 30 18:52:23.162249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.162226 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-logging/data-science-cluster-log-forwarder-zgmkr"] Oct 30 18:52:23.169123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.169102 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:52:23.182677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182658 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpvsw\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-kube-api-access-tpvsw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182679 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182690 6966 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-lokistack-gateway\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182699 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182709 6966 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tenants\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182717 6966 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/74cb7faa-cb74-4097-aa10-9d4b9c474a10-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182726 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/74cb7faa-cb74-4097-aa10-9d4b9c474a10-data-science-lokistack-gateway-client-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.182754 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.182734 6966 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/74cb7faa-cb74-4097-aa10-9d4b9c474a10-tls-secret\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.196241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.196215 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf"] Oct 30 18:52:23.200784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.200755 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-gateway-6f9b9f4d87-svslf"] Oct 30 18:52:23.283515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283493 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283603 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283671 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283747 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283829 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283797 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225\" (UniqueName: \"kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283832 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283866 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http\") pod \"3b1af291-85bb-4df5-9b8b-9c38baa54023\" (UID: \"3b1af291-85bb-4df5-9b8b-9c38baa54023\") " Oct 30 18:52:23.283987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.283898 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.284257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.284086 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "data-science-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:23.284447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.284424 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.284831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.284571 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.286208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.286184 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "kube-api-access-t7225". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:23.286208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.286199 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "data-science-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.286350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.286251 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:52:23.286415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.286395 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "data-science-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.286502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.286452 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret" pod "3b1af291-85bb-4df5-9b8b-9c38baa54023" (UID: "3b1af291-85bb-4df5-9b8b-9c38baa54023"). InnerVolumeSpecName "test-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:23.288519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.288496 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:23.385218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.385161 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-grpc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.385218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.385185 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7225\" (UniqueName: \"kubernetes.io/projected/3b1af291-85bb-4df5-9b8b-9c38baa54023-kube-api-access-t7225\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.385218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.385196 6966 reconciler_common.go:299] "Volume detached for volume \"test-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-test-logs-loki-secret\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.385218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.385204 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/3b1af291-85bb-4df5-9b8b-9c38baa54023-data-science-lokistack-compactor-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:23.385414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.385231 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:52:23.395244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.395229 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:52:23.411929 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.411906 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcb1827-cc31-4c3c-a68e-d69ddc40db03" path="/var/lib/kubelet/pods/0bcb1827-cc31-4c3c-a68e-d69ddc40db03/volumes" Oct 30 18:52:23.412327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.412314 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24b54b4d-390a-4331-9b0a-551834333310" path="/var/lib/kubelet/pods/24b54b4d-390a-4331-9b0a-551834333310/volumes" Oct 30 18:52:23.412697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.412685 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2fbe6c-cedf-4f1e-a19d-fdb918247882" path="/var/lib/kubelet/pods/6c2fbe6c-cedf-4f1e-a19d-fdb918247882/volumes" Oct 30 18:52:23.413308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.413295 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" path="/var/lib/kubelet/pods/74cb7faa-cb74-4097-aa10-9d4b9c474a10/volumes" Oct 30 18:52:23.485848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:23.485827 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-b0de9d3a-bff5-4ef4-9e95-b9b277a634b5\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-02d0b178f74a0edb2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-6112057b00a5460799d60e94c2b0618d4339163956b487ae0109084bbd0cec7d\"" Oct 30 18:52:24.107802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:24.107772 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" Oct 30 18:52:24.108246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:24.107798 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-lokistack-compactor-0" event={"ID":"3b1af291-85bb-4df5-9b8b-9c38baa54023","Type":"ContainerDied","Data":"45d61954326f6f4738da52d5904dadad3ac7c3a2e69226eff9fa6a864912d5ec"} Oct 30 18:52:24.108246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:24.107849 6966 scope.go:122] "RemoveContainer" containerID="b1336791803486eaa3d921dfdf0c7fe8583f7f06139366a267f446e3eebb7a77" Oct 30 18:52:24.159670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:24.159637 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 18:52:24.171023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:24.171000 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-lokistack-compactor-0"] Oct 30 18:52:24.279817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:24.279790 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:25.411765 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:25.411729 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b1af291-85bb-4df5-9b8b-9c38baa54023" path="/var/lib/kubelet/pods/3b1af291-85bb-4df5-9b8b-9c38baa54023/volumes" Oct 30 18:52:30.254373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254339 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s"] Oct 30 18:52:30.254832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254741 6966 cpu_manager.go:414] "containerMap: removing container" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="opa" Oct 30 18:52:30.254832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254759 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="opa" Oct 30 18:52:30.254832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254786 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0bcb1827-cc31-4c3c-a68e-d69ddc40db03" containerName="korrel8r" Oct 30 18:52:30.254832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254795 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcb1827-cc31-4c3c-a68e-d69ddc40db03" containerName="korrel8r" Oct 30 18:52:30.254832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254823 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6c2fbe6c-cedf-4f1e-a19d-fdb918247882" containerName="loki-query-frontend" Oct 30 18:52:30.254832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254829 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2fbe6c-cedf-4f1e-a19d-fdb918247882" containerName="loki-query-frontend" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254848 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3b1af291-85bb-4df5-9b8b-9c38baa54023" containerName="loki-compactor" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254855 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b1af291-85bb-4df5-9b8b-9c38baa54023" containerName="loki-compactor" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254870 6966 cpu_manager.go:414] "containerMap: removing container" podUID="24b54b4d-390a-4331-9b0a-551834333310" containerName="collector" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254875 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24b54b4d-390a-4331-9b0a-551834333310" containerName="collector" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254884 6966 cpu_manager.go:414] "containerMap: removing container" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="gateway" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254891 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="gateway" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254977 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="opa" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.254989 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcb1827-cc31-4c3c-a68e-d69ddc40db03" containerName="korrel8r" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.255002 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2fbe6c-cedf-4f1e-a19d-fdb918247882" containerName="loki-query-frontend" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.255014 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b1af291-85bb-4df5-9b8b-9c38baa54023" containerName="loki-compactor" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.255023 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="24b54b4d-390a-4331-9b0a-551834333310" containerName="collector" Oct 30 18:52:30.255128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.255033 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="74cb7faa-cb74-4097-aa10-9d4b9c474a10" containerName="gateway" Oct 30 18:52:30.259713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.259694 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.285171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.285139 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 18:52:30.285298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.285199 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rl5v4\"" Oct 30 18:52:30.285422 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.285402 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 18:52:30.287346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.287076 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s"] Oct 30 18:52:30.337305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.337283 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2k75\" (UniqueName: \"kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.337419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.337320 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.337479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.337437 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.438139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.438111 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.438721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.438701 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2k75\" (UniqueName: \"kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.438834 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.438739 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.439294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.439272 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.440692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.440673 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.460944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.460918 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2k75\" (UniqueName: \"kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75\") pod \"data-science-usage-logs-collector-76bd859946-b6f5s\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.497415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.497379 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:30.572687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.572657 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:30.724530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.724495 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:30.880524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.880417 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s"] Oct 30 18:52:30.885875 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:52:30.885849 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc72ea0b0_9671_4f73_aa8c_79c2f7e76959.slice/crio-4e6d26e49b50fc50058002c93110795b608fa42cec19f13e61318cb25a0f965b WatchSource:0}: Error finding container 4e6d26e49b50fc50058002c93110795b608fa42cec19f13e61318cb25a0f965b: Status 404 returned error can't find the container with id 4e6d26e49b50fc50058002c93110795b608fa42cec19f13e61318cb25a0f965b Oct 30 18:52:30.888405 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.888370 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:52:30.888501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:30.888429 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:52:31.154490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:31.154351 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" event={"ID":"c72ea0b0-9671-4f73-aa8c-79c2f7e76959","Type":"ContainerStarted","Data":"e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb"} Oct 30 18:52:31.154490 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:31.154405 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" event={"ID":"c72ea0b0-9671-4f73-aa8c-79c2f7e76959","Type":"ContainerStarted","Data":"4e6d26e49b50fc50058002c93110795b608fa42cec19f13e61318cb25a0f965b"} Oct 30 18:52:31.194710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:31.194637 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" podStartSLOduration=1.194618792 podStartE2EDuration="1.194618792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:31.193238122 +0000 UTC m=+3313.899423479" watchObservedRunningTime="2026-10-30 18:52:31.194618792 +0000 UTC m=+3313.900804151" Oct 30 18:52:32.770926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.770880 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:32.796005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.795976 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc"] Oct 30 18:52:32.796168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.796141 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" podUID="df1b6f47-2b39-4531-b7ce-88bf1b301a79" containerName="loki-query-frontend" containerID="cri-o://b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8" gracePeriod=30 Oct 30 18:52:32.804442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.804412 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b"] Oct 30 18:52:32.804669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.804645 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="gateway" containerID="cri-o://53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1" gracePeriod=30 Oct 30 18:52:32.804843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.804822 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="opa" containerID="cri-o://65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129" gracePeriod=30 Oct 30 18:52:32.813221 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.813200 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm"] Oct 30 18:52:32.907176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.907144 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s"] Oct 30 18:52:32.997914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:32.997868 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:33.056529 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.056509 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:52:33.164955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.164925 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " Oct 30 18:52:33.165083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.164965 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config\") pod \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " Oct 30 18:52:33.165083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.165022 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http\") pod \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " Oct 30 18:52:33.165083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.165045 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl\" (UniqueName: \"kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl\") pod \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " Oct 30 18:52:33.165083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.165075 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle\") pod \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\" (UID: \"df1b6f47-2b39-4531-b7ce-88bf1b301a79\") " Oct 30 18:52:33.165571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.165542 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle" pod "df1b6f47-2b39-4531-b7ce-88bf1b301a79" (UID: "df1b6f47-2b39-4531-b7ce-88bf1b301a79"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:33.165813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.165790 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config" pod "df1b6f47-2b39-4531-b7ce-88bf1b301a79" (UID: "df1b6f47-2b39-4531-b7ce-88bf1b301a79"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:33.167096 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.167067 6966 generic.go:356] "Generic (PLEG): container finished" podID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerID="65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129" exitCode=0 Oct 30 18:52:33.167198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.167124 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" event={"ID":"09ed9570-561f-4c0c-828d-ca47342e2fd4","Type":"ContainerDied","Data":"65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129"} Oct 30 18:52:33.167424 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.167390 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc" pod "df1b6f47-2b39-4531-b7ce-88bf1b301a79" (UID: "df1b6f47-2b39-4531-b7ce-88bf1b301a79"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.167760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.167738 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl" pod "df1b6f47-2b39-4531-b7ce-88bf1b301a79" (UID: "df1b6f47-2b39-4531-b7ce-88bf1b301a79"). InnerVolumeSpecName "kube-api-access-fvtnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:33.167848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.167765 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http" pod "df1b6f47-2b39-4531-b7ce-88bf1b301a79" (UID: "df1b6f47-2b39-4531-b7ce-88bf1b301a79"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.168640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.168614 6966 generic.go:356] "Generic (PLEG): container finished" podID="df1b6f47-2b39-4531-b7ce-88bf1b301a79" containerID="b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8" exitCode=0 Oct 30 18:52:33.168724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.168653 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" Oct 30 18:52:33.168724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.168686 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" event={"ID":"df1b6f47-2b39-4531-b7ce-88bf1b301a79","Type":"ContainerDied","Data":"b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8"} Oct 30 18:52:33.168820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.168727 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc" event={"ID":"df1b6f47-2b39-4531-b7ce-88bf1b301a79","Type":"ContainerDied","Data":"43416850c6f42e0cb96aafc57bd8f8f08f23cbd6efaf4576944005d80a355760"} Oct 30 18:52:33.168820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.168743 6966 scope.go:122] "RemoveContainer" containerID="b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8" Oct 30 18:52:33.168982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.168957 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" podUID="d08f6449-219f-48a5-bae4-3453c806d4df" containerName="loki-querier" containerID="cri-o://6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668" gracePeriod=30 Oct 30 18:52:33.169090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.169023 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" podUID="c72ea0b0-9671-4f73-aa8c-79c2f7e76959" containerName="otc-container" containerID="cri-o://e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb" gracePeriod=30 Oct 30 18:52:33.181500 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.181483 6966 scope.go:122] "RemoveContainer" containerID="b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8" Oct 30 18:52:33.181775 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:33.181755 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8\": container with ID starting with b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8 not found: ID does not exist" containerID="b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8" Oct 30 18:52:33.181822 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.181783 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8"} err="failed to get container status \"b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8\": rpc error: code = NotFound desc = could not find container \"b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8\": container with ID starting with b45f91cb1f2bf6d3c98253f5e5af79a4079af34c7c5ed8e67cca537d6ab7e9c8 not found: ID does not exist" Oct 30 18:52:33.187056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.187024 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" podUID="d08f6449-219f-48a5-bae4-3453c806d4df" containerName="loki-querier" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:52:33.218876 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.218855 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc"] Oct 30 18:52:33.221531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.221512 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbwc4xc"] Oct 30 18:52:33.234286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.234256 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:33.266134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.266110 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-grpc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.266134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.266131 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.266253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.266141 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-query-frontend-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.266253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.266151 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtnl\" (UniqueName: \"kubernetes.io/projected/df1b6f47-2b39-4531-b7ce-88bf1b301a79-kube-api-access-fvtnl\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.266253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.266160 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df1b6f47-2b39-4531-b7ce-88bf1b301a79-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.410477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.410440 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:33.412998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.412975 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1b6f47-2b39-4531-b7ce-88bf1b301a79" path="/var/lib/kubelet/pods/df1b6f47-2b39-4531-b7ce-88bf1b301a79/volumes" Oct 30 18:52:33.468593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.468566 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75\" (UniqueName: \"kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75\") pod \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " Oct 30 18:52:33.468708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.468658 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs\") pod \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " Oct 30 18:52:33.468748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.468721 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal\" (UniqueName: \"kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal\") pod \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\" (UID: \"c72ea0b0-9671-4f73-aa8c-79c2f7e76959\") " Oct 30 18:52:33.469082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.469060 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal" pod "c72ea0b0-9671-4f73-aa8c-79c2f7e76959" (UID: "c72ea0b0-9671-4f73-aa8c-79c2f7e76959"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:33.470651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.470628 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75" pod "c72ea0b0-9671-4f73-aa8c-79c2f7e76959" (UID: "c72ea0b0-9671-4f73-aa8c-79c2f7e76959"). InnerVolumeSpecName "kube-api-access-h2k75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:33.470743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.470685 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs" pod "c72ea0b0-9671-4f73-aa8c-79c2f7e76959" (UID: "c72ea0b0-9671-4f73-aa8c-79c2f7e76959"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:33.569540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.569518 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.569540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.569538 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2k75\" (UniqueName: \"kubernetes.io/projected/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-kube-api-access-h2k75\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:33.569675 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:33.569549 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/c72ea0b0-9671-4f73-aa8c-79c2f7e76959-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.052553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.052529 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:52:34.174710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.174642 6966 generic.go:356] "Generic (PLEG): container finished" podID="c72ea0b0-9671-4f73-aa8c-79c2f7e76959" containerID="e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb" exitCode=0 Oct 30 18:52:34.174710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.174684 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" Oct 30 18:52:34.174916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.174725 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" event={"ID":"c72ea0b0-9671-4f73-aa8c-79c2f7e76959","Type":"ContainerDied","Data":"e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb"} Oct 30 18:52:34.174916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.174775 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s" event={"ID":"c72ea0b0-9671-4f73-aa8c-79c2f7e76959","Type":"ContainerDied","Data":"4e6d26e49b50fc50058002c93110795b608fa42cec19f13e61318cb25a0f965b"} Oct 30 18:52:34.174916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.174803 6966 scope.go:122] "RemoveContainer" containerID="e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb" Oct 30 18:52:34.175672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175622 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.175672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175667 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.175901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175702 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.175901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175738 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.175901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175768 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.175901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175803 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.175901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175880 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.176241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.175931 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http\") pod \"09ed9570-561f-4c0c-828d-ca47342e2fd4\" (UID: \"09ed9570-561f-4c0c-828d-ca47342e2fd4\") " Oct 30 18:52:34.178304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.178274 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.178870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.178837 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.179047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.179025 6966 generic.go:356] "Generic (PLEG): container finished" podID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerID="53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1" exitCode=0 Oct 30 18:52:34.179128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.179113 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" event={"ID":"09ed9570-561f-4c0c-828d-ca47342e2fd4","Type":"ContainerDied","Data":"53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1"} Oct 30 18:52:34.179195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.179147 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" event={"ID":"09ed9570-561f-4c0c-828d-ca47342e2fd4","Type":"ContainerDied","Data":"c767ee23fd9f8382639fa815f05ee4eeddecae14942a779dba779fc40ec87a81"} Oct 30 18:52:34.179259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.179220 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b" Oct 30 18:52:34.179259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.179243 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:34.179626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.179594 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "kube-api-access-sq2h7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.182912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.182860 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:34.183077 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.183056 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.183767 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.183748 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:34.187154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.187131 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants" pod "09ed9570-561f-4c0c-828d-ca47342e2fd4" (UID: "09ed9570-561f-4c0c-828d-ca47342e2fd4"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:34.191564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.191537 6966 scope.go:122] "RemoveContainer" containerID="e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb" Oct 30 18:52:34.191793 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:34.191773 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb\": container with ID starting with e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb not found: ID does not exist" containerID="e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb" Oct 30 18:52:34.191831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.191802 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb"} err="failed to get container status \"e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb\": rpc error: code = NotFound desc = could not find container \"e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb\": container with ID starting with e686352d9ca18d226015ab3c5699235918cd6c03b3d5435ece7d7087863898fb not found: ID does not exist" Oct 30 18:52:34.191831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.191818 6966 scope.go:122] "RemoveContainer" containerID="65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129" Oct 30 18:52:34.199507 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.199450 6966 scope.go:122] "RemoveContainer" containerID="53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1" Oct 30 18:52:34.207455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.207439 6966 scope.go:122] "RemoveContainer" containerID="65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129" Oct 30 18:52:34.207717 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:34.207694 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129\": container with ID starting with 65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129 not found: ID does not exist" containerID="65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129" Oct 30 18:52:34.207779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.207716 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129"} err="failed to get container status \"65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129\": rpc error: code = NotFound desc = could not find container \"65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129\": container with ID starting with 65dbe37539c3372c20055e07e0f5454acb62cc3c4891b1aa8f18022b3b33c129 not found: ID does not exist" Oct 30 18:52:34.207779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.207735 6966 scope.go:122] "RemoveContainer" containerID="53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1" Oct 30 18:52:34.207990 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:34.207971 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1\": container with ID starting with 53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1 not found: ID does not exist" containerID="53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1" Oct 30 18:52:34.208048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.207998 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1"} err="failed to get container status \"53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1\": rpc error: code = NotFound desc = could not find container \"53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1\": container with ID starting with 53fbb7e873830b7f14f5625fa2e425817af36e337e3bd264527cd097c9b1f2c1 not found: ID does not exist" Oct 30 18:52:34.236244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.236223 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s"] Oct 30 18:52:34.243560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.243539 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-76bd859946-b6f5s"] Oct 30 18:52:34.276580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276558 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276577 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276588 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276599 6966 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/09ed9570-561f-4c0c-828d-ca47342e2fd4-tenants\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276607 6966 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276615 6966 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/09ed9570-561f-4c0c-828d-ca47342e2fd4-lokistack-gateway\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276624 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sq2h7\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-kube-api-access-sq2h7\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.276701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.276632 6966 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/09ed9570-561f-4c0c-828d-ca47342e2fd4-tls-secret\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:34.522286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.522260 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b"] Oct 30 18:52:34.533376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:34.533356 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-7699f488fb-hpc6b"] Oct 30 18:52:35.411731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:35.411699 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" path="/var/lib/kubelet/pods/09ed9570-561f-4c0c-828d-ca47342e2fd4/volumes" Oct 30 18:52:35.412125 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:35.412110 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c72ea0b0-9671-4f73-aa8c-79c2f7e76959" path="/var/lib/kubelet/pods/c72ea0b0-9671-4f73-aa8c-79c2f7e76959/volumes" Oct 30 18:52:38.464008 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.463976 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.600130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600105 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:52:38.600485 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600470 6966 cpu_manager.go:414] "containerMap: removing container" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="gateway" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600487 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="gateway" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600504 6966 cpu_manager.go:414] "containerMap: removing container" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="opa" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600509 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="opa" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600536 6966 cpu_manager.go:414] "containerMap: removing container" podUID="df1b6f47-2b39-4531-b7ce-88bf1b301a79" containerName="loki-query-frontend" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600543 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1b6f47-2b39-4531-b7ce-88bf1b301a79" containerName="loki-query-frontend" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600553 6966 cpu_manager.go:414] "containerMap: removing container" podUID="c72ea0b0-9671-4f73-aa8c-79c2f7e76959" containerName="otc-container" Oct 30 18:52:38.600590 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600560 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c72ea0b0-9671-4f73-aa8c-79c2f7e76959" containerName="otc-container" Oct 30 18:52:38.601058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600643 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1b6f47-2b39-4531-b7ce-88bf1b301a79" containerName="loki-query-frontend" Oct 30 18:52:38.601058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600679 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="c72ea0b0-9671-4f73-aa8c-79c2f7e76959" containerName="otc-container" Oct 30 18:52:38.601058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600692 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="gateway" Oct 30 18:52:38.601058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.600701 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ed9570-561f-4c0c-828d-ca47342e2fd4" containerName="opa" Oct 30 18:52:38.606391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.606372 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.631092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.631067 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:52:38.638053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.638029 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"test-loki-lifecycle-secret\"" Oct 30 18:52:38.638162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.638122 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-http\"" Oct 30 18:52:38.638162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.638128 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-dockercfg-hb59n\"" Oct 30 18:52:38.638162 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.638121 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-compactor-grpc\"" Oct 30 18:52:38.681117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.681095 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg"] Oct 30 18:52:38.684781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.684761 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.700173 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.700146 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.705926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.705899 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg"] Oct 30 18:52:38.709545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.709524 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.709627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.709555 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.709627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.709577 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxbdj\" (UniqueName: \"kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.709731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.709656 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.709731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.709708 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.709803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.709785 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.738299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.738247 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-http\"" Oct 30 18:52:38.738299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.738247 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-query-frontend-grpc\"" Oct 30 18:52:38.810589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810559 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.810731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810595 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.810731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810618 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxbdj\" (UniqueName: \"kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.810731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810683 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.810731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810707 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.810731 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810725 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.810934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810758 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6jzz\" (UniqueName: \"kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.810934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810906 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.811035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.810937 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.811137 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.811112 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.811397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.811367 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.811573 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.811404 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.812264 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.812237 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.813440 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.813418 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.813566 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.813537 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.814060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.814035 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.832996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.832977 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxbdj\" (UniqueName: \"kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:38.912561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.912525 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.912688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.912568 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.912688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.912621 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.912688 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.912652 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.912823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.912792 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6jzz\" (UniqueName: \"kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.913308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.913284 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.913426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.913379 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.914993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.914972 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.915060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.915030 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.928390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.928362 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:38.937926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.937901 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6jzz\" (UniqueName: \"kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz\") pod \"data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:38.971670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.971650 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v"] Oct 30 18:52:38.978950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.978885 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:38.995802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.995768 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-ca-bundle\"" Oct 30 18:52:38.995901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.995833 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:52:38.997058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:38.997034 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:39.000167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.000134 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v"] Oct 30 18:52:39.014257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014223 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014370 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014319 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014374 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014546 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014527 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014561 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g847q\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014630 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014728 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.014792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.014782 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.067718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.067179 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-http\"" Oct 30 18:52:39.067718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.067291 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway\"" Oct 30 18:52:39.067718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.067376 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-dockercfg-bntg7\"" Oct 30 18:52:39.067718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.067481 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-lokistack-gateway-client-http\"" Oct 30 18:52:39.115817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.115783 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.115973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.115835 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.115973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.115883 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.115973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.115934 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.115973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.115966 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g847q\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.116226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.115996 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.116226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.116046 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.116226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.116093 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.116995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.116958 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.117642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.117606 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.117798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.117771 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.118498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.118477 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.118990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.118970 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.119808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.119788 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.119966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.119949 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.136761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.136739 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:52:39.140139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.140119 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g847q\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q\") pod \"data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.164998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.164963 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:39.214256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.214228 6966 generic.go:356] "Generic (PLEG): container finished" podID="d08f6449-219f-48a5-bae4-3453c806d4df" containerID="6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668" exitCode=0 Oct 30 18:52:39.214530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.214273 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" Oct 30 18:52:39.214530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.214275 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" event={"ID":"d08f6449-219f-48a5-bae4-3453c806d4df","Type":"ContainerDied","Data":"6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668"} Oct 30 18:52:39.214530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.214356 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm" event={"ID":"d08f6449-219f-48a5-bae4-3453c806d4df","Type":"ContainerDied","Data":"7662b1b4f88cdd55385fb4a70c551244cf3fd66d40d65e975bba979f9c05c3de"} Oct 30 18:52:39.214530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.214389 6966 scope.go:122] "RemoveContainer" containerID="6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668" Oct 30 18:52:39.216642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.216622 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt\" (UniqueName: \"kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt\") pod \"d08f6449-219f-48a5-bae4-3453c806d4df\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " Oct 30 18:52:39.216757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.216703 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc\") pod \"d08f6449-219f-48a5-bae4-3453c806d4df\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " Oct 30 18:52:39.216832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.216783 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config\") pod \"d08f6449-219f-48a5-bae4-3453c806d4df\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " Oct 30 18:52:39.216832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.216822 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret\") pod \"d08f6449-219f-48a5-bae4-3453c806d4df\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " Oct 30 18:52:39.216944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.216857 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle\") pod \"d08f6449-219f-48a5-bae4-3453c806d4df\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " Oct 30 18:52:39.216944 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.216914 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http\") pod \"d08f6449-219f-48a5-bae4-3453c806d4df\" (UID: \"d08f6449-219f-48a5-bae4-3453c806d4df\") " Oct 30 18:52:39.217535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.217505 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle" pod "d08f6449-219f-48a5-bae4-3453c806d4df" (UID: "d08f6449-219f-48a5-bae4-3453c806d4df"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:39.217654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.217607 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config" pod "d08f6449-219f-48a5-bae4-3453c806d4df" (UID: "d08f6449-219f-48a5-bae4-3453c806d4df"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:52:39.219181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.219152 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc" pod "d08f6449-219f-48a5-bae4-3453c806d4df" (UID: "d08f6449-219f-48a5-bae4-3453c806d4df"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:39.219291 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.219193 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http" pod "d08f6449-219f-48a5-bae4-3453c806d4df" (UID: "d08f6449-219f-48a5-bae4-3453c806d4df"). InnerVolumeSpecName "data-science-usage-logs-lokistack-querier-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:39.219453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.219429 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret" pod "d08f6449-219f-48a5-bae4-3453c806d4df" (UID: "d08f6449-219f-48a5-bae4-3453c806d4df"). InnerVolumeSpecName "test-usage-logs-loki-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:52:39.219697 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.219676 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt" pod "d08f6449-219f-48a5-bae4-3453c806d4df" (UID: "d08f6449-219f-48a5-bae4-3453c806d4df"). InnerVolumeSpecName "kube-api-access-4f7gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:52:39.224144 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.224125 6966 scope.go:122] "RemoveContainer" containerID="6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668" Oct 30 18:52:39.224433 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:39.224407 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668\": container with ID starting with 6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668 not found: ID does not exist" containerID="6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668" Oct 30 18:52:39.224549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.224443 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668"} err="failed to get container status \"6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668\": rpc error: code = NotFound desc = could not find container \"6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668\": container with ID starting with 6d89832ae376768c83035b83a11fe2c8d1de3ed66d8753b2133b3170bef35668 not found: ID does not exist" Oct 30 18:52:39.296710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.296655 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:39.318253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.318225 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f7gt\" (UniqueName: \"kubernetes.io/projected/d08f6449-219f-48a5-bae4-3453c806d4df-kube-api-access-4f7gt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:39.318347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.318261 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-grpc\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-grpc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:39.318347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.318279 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:39.318347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.318294 6966 reconciler_common.go:299] "Volume detached for volume \"test-usage-logs-loki-secret\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-test-usage-logs-loki-secret\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:39.318347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.318307 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:39.318347 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.318322 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-querier-http\" (UniqueName: \"kubernetes.io/secret/d08f6449-219f-48a5-bae4-3453c806d4df-data-science-usage-logs-lokistack-querier-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:52:39.331582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.331550 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg"] Oct 30 18:52:39.384250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.384217 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:52:39.558803 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.558678 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm"] Oct 30 18:52:39.563968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.563943 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-querier-8577fdf8bc-8p4dm"] Oct 30 18:52:39.603282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:39.603256 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v"] Oct 30 18:52:39.608474 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:52:39.608431 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfe0d3020_1746_42eb_b9f0_ff9127444f18.slice/crio-df8e7e7425c2d1fffa9c814175a2f1351f5a511e9b94e642249f4ea3ebebf243 WatchSource:0}: Error finding container df8e7e7425c2d1fffa9c814175a2f1351f5a511e9b94e642249f4ea3ebebf243: Status 404 returned error can't find the container with id df8e7e7425c2d1fffa9c814175a2f1351f5a511e9b94e642249f4ea3ebebf243 Oct 30 18:52:40.222234 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.222132 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" event={"ID":"34f216a3-cccd-44ee-8e4b-65d7436c4e08","Type":"ContainerStarted","Data":"1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff"} Oct 30 18:52:40.222234 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.222195 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:52:40.222234 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.222216 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" event={"ID":"34f216a3-cccd-44ee-8e4b-65d7436c4e08","Type":"ContainerStarted","Data":"80f43a3d918120b150bbcfce6cb4667a41b2a1884101383a0156706c38e423e3"} Oct 30 18:52:40.223917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.223890 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" event={"ID":"fe0d3020-1746-42eb-b9f0-ff9127444f18","Type":"ContainerStarted","Data":"e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d"} Oct 30 18:52:40.224049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.223924 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" event={"ID":"fe0d3020-1746-42eb-b9f0-ff9127444f18","Type":"ContainerStarted","Data":"ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b"} Oct 30 18:52:40.224049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.223936 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" event={"ID":"fe0d3020-1746-42eb-b9f0-ff9127444f18","Type":"ContainerStarted","Data":"df8e7e7425c2d1fffa9c814175a2f1351f5a511e9b94e642249f4ea3ebebf243"} Oct 30 18:52:40.224049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.224016 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:40.224049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.224037 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:40.230319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.230297 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:40.233332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.233313 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:52:40.263002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:40.262962 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" podStartSLOduration=2.262950328 podStartE2EDuration="2.262950328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:40.261522224 +0000 UTC m=+3322.967707580" watchObservedRunningTime="2026-10-30 18:52:40.262950328 +0000 UTC m=+3322.969135684" Oct 30 18:52:41.411330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:41.411298 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d08f6449-219f-48a5-bae4-3453c806d4df" path="/var/lib/kubelet/pods/d08f6449-219f-48a5-bae4-3453c806d4df/volumes" Oct 30 18:52:46.779867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:46.779832 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:46.807205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:46.807175 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") device path: \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:46.880540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:46.880517 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:46.880664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:46.880648 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") DevicePath \"\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:46.894301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:46.894280 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") DevicePath \"csi-908cbe7987bf0ef7acb0c43ba5742dca3eade156d34ceb83ea4b9dc800a9dca5\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:46.994249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:46.994221 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/90f68065c219b046956a3eedc2bf9d54a167b2b54b7bb678301a6348f6107900/globalmount\"" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:47.019538 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:47.019516 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"data-science-usage-logs-lokistack-compactor-0\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:47.198539 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:52:47.198507 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:52:47.320036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:47.320006 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:47.624168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:47.624118 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" podStartSLOduration=9.624105837 podStartE2EDuration="9.624105837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:40.308712819 +0000 UTC m=+3323.014898176" watchObservedRunningTime="2026-10-30 18:52:47.624105837 +0000 UTC m=+3330.330291193" Oct 30 18:52:47.624722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:47.624696 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:52:47.629553 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:52:47.629524 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6429ea9c_b300_4a44_897c_fa800c6d8ccc.slice/crio-86701ae9a33cda4f7a44cd1766b726147e50fcc9e4b8376ea20bd8de90367203 WatchSource:0}: Error finding container 86701ae9a33cda4f7a44cd1766b726147e50fcc9e4b8376ea20bd8de90367203: Status 404 returned error can't find the container with id 86701ae9a33cda4f7a44cd1766b726147e50fcc9e4b8376ea20bd8de90367203 Oct 30 18:52:48.266071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:48.266033 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"6429ea9c-b300-4a44-897c-fa800c6d8ccc","Type":"ContainerStarted","Data":"d7ee918e84dfec2637fc436a9ff713ac96708c976c063705a9903ed663f82cf3"} Oct 30 18:52:48.266450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:48.266083 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:52:48.266450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:48.266096 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"6429ea9c-b300-4a44-897c-fa800c6d8ccc","Type":"ContainerStarted","Data":"86701ae9a33cda4f7a44cd1766b726147e50fcc9e4b8376ea20bd8de90367203"} Oct 30 18:52:48.308229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:52:48.308183 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podStartSLOduration=10.308171812 podStartE2EDuration="10.308171812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:52:48.307454465 +0000 UTC m=+3331.013639821" watchObservedRunningTime="2026-10-30 18:52:48.308171812 +0000 UTC m=+3331.014357168" Oct 30 18:53:01.235242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:01.235168 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:53:09.277620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:09.277591 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:53:17.199681 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:53:17.199634 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:53:23.420068 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:23.420030 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:25.590402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:25.590364 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:43.171361 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.171328 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx"] Oct 30 18:53:43.171878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.171731 6966 cpu_manager.go:414] "containerMap: removing container" podUID="d08f6449-219f-48a5-bae4-3453c806d4df" containerName="loki-querier" Oct 30 18:53:43.171878 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.171746 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d08f6449-219f-48a5-bae4-3453c806d4df" containerName="loki-querier" Oct 30 18:53:43.172000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.171907 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="d08f6449-219f-48a5-bae4-3453c806d4df" containerName="loki-querier" Oct 30 18:53:43.175380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.175360 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.201523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.201496 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-headless-tls\"" Oct 30 18:53:43.201685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.201544 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-dockercfg-rqrn7\"" Oct 30 18:53:43.203848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.203825 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx"] Oct 30 18:53:43.221749 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.221722 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.221853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.221819 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.221915 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.221900 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7qw4\" (UniqueName: \"kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.272119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.272095 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-usage-logs-collector-9d8884b6\"" Oct 30 18:53:43.314045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.314014 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:43.323080 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.323059 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.323172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.323125 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7qw4\" (UniqueName: \"kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.323172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.323158 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.323694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.323669 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.325552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.325526 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.359821 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.359793 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7qw4\" (UniqueName: \"kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4\") pod \"data-science-usage-logs-collector-74fd5665b8-gr2mx\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.490662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.490607 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:43.596792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.596747 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:43.787837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.787733 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx"] Oct 30 18:53:43.792220 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:53:43.792193 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27b2870_43c0_4885_8208_6b0b9998e447.slice/crio-7db911813698cefb97d34a565e45ac5a7881bff37e05f10cd043e120e8dd33e3 WatchSource:0}: Error finding container 7db911813698cefb97d34a565e45ac5a7881bff37e05f10cd043e120e8dd33e3: Status 404 returned error can't find the container with id 7db911813698cefb97d34a565e45ac5a7881bff37e05f10cd043e120e8dd33e3 Oct 30 18:53:43.794657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.794624 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:53:43.794763 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:43.794693 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:53:44.552183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:44.552147 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" event={"ID":"f27b2870-43c0-4885-8208-6b0b9998e447","Type":"ContainerStarted","Data":"db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9"} Oct 30 18:53:44.552612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:44.552211 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" event={"ID":"f27b2870-43c0-4885-8208-6b0b9998e447","Type":"ContainerStarted","Data":"7db911813698cefb97d34a565e45ac5a7881bff37e05f10cd043e120e8dd33e3"} Oct 30 18:53:44.591873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:44.591827 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" podStartSLOduration=1.591816112 podStartE2EDuration="1.591816112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:44.591151548 +0000 UTC m=+3387.297336906" watchObservedRunningTime="2026-10-30 18:53:44.591816112 +0000 UTC m=+3387.298001469" Oct 30 18:53:46.755614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.755580 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:53:46.756100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.755930 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" podUID="6429ea9c-b300-4a44-897c-fa800c6d8ccc" containerName="loki-compactor" containerID="cri-o://d7ee918e84dfec2637fc436a9ff713ac96708c976c063705a9903ed663f82cf3" gracePeriod=30 Oct 30 18:53:46.768429 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.768400 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:46.818840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.818808 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg"] Oct 30 18:53:46.819233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.819198 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" podUID="34f216a3-cccd-44ee-8e4b-65d7436c4e08" containerName="loki-query-frontend" containerID="cri-o://1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff" gracePeriod=30 Oct 30 18:53:46.845448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.845425 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v"] Oct 30 18:53:46.845668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.845643 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="gateway" containerID="cri-o://ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b" gracePeriod=30 Oct 30 18:53:46.845668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.845656 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="opa" containerID="cri-o://e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d" gracePeriod=30 Oct 30 18:53:46.912736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.910672 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx"] Oct 30 18:53:46.912736 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.911496 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" podUID="f27b2870-43c0-4885-8208-6b0b9998e447" containerName="otc-container" containerID="cri-o://db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9" gracePeriod=30 Oct 30 18:53:46.992596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:46.992567 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:47.090201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.090178 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:53:47.151369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.151344 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:47.152315 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152296 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle\") pod \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " Oct 30 18:53:47.152376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152333 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz\" (UniqueName: \"kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz\") pod \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " Oct 30 18:53:47.152376 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152363 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc\") pod \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " Oct 30 18:53:47.152488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152452 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http\") pod \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " Oct 30 18:53:47.152563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152548 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config\") pod \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\" (UID: \"34f216a3-cccd-44ee-8e4b-65d7436c4e08\") " Oct 30 18:53:47.152705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152667 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle" pod "34f216a3-cccd-44ee-8e4b-65d7436c4e08" (UID: "34f216a3-cccd-44ee-8e4b-65d7436c4e08"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.152975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152951 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.152975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.152952 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config" pod "34f216a3-cccd-44ee-8e4b-65d7436c4e08" (UID: "34f216a3-cccd-44ee-8e4b-65d7436c4e08"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.154446 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.154423 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz" pod "34f216a3-cccd-44ee-8e4b-65d7436c4e08" (UID: "34f216a3-cccd-44ee-8e4b-65d7436c4e08"). InnerVolumeSpecName "kube-api-access-c6jzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.154611 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.154589 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc" pod "34f216a3-cccd-44ee-8e4b-65d7436c4e08" (UID: "34f216a3-cccd-44ee-8e4b-65d7436c4e08"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.154840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.154813 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http" pod "34f216a3-cccd-44ee-8e4b-65d7436c4e08" (UID: "34f216a3-cccd-44ee-8e4b-65d7436c4e08"). InnerVolumeSpecName "data-science-usage-logs-lokistack-query-frontend-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.199013 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:53:47.198982 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:53:47.217065 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.217030 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:53:47.253233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253212 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs\") pod \"f27b2870-43c0-4885-8208-6b0b9998e447\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " Oct 30 18:53:47.253357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253247 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal\" (UniqueName: \"kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal\") pod \"f27b2870-43c0-4885-8208-6b0b9998e447\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " Oct 30 18:53:47.253357 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253272 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4\" (UniqueName: \"kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4\") pod \"f27b2870-43c0-4885-8208-6b0b9998e447\" (UID: \"f27b2870-43c0-4885-8208-6b0b9998e447\") " Oct 30 18:53:47.253528 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253508 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6jzz\" (UniqueName: \"kubernetes.io/projected/34f216a3-cccd-44ee-8e4b-65d7436c4e08-kube-api-access-c6jzz\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.253579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253535 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-grpc\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-grpc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.253579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253554 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-query-frontend-http\" (UniqueName: \"kubernetes.io/secret/34f216a3-cccd-44ee-8e4b-65d7436c4e08-data-science-usage-logs-lokistack-query-frontend-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.253579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253570 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/34f216a3-cccd-44ee-8e4b-65d7436c4e08-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.253721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.253676 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal" pod "f27b2870-43c0-4885-8208-6b0b9998e447" (UID: "f27b2870-43c0-4885-8208-6b0b9998e447"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.255187 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.255165 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs" pod "f27b2870-43c0-4885-8208-6b0b9998e447" (UID: "f27b2870-43c0-4885-8208-6b0b9998e447"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.255279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.255256 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4" pod "f27b2870-43c0-4885-8208-6b0b9998e447" (UID: "f27b2870-43c0-4885-8208-6b0b9998e447"). InnerVolumeSpecName "kube-api-access-l7qw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.354298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.354277 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/f27b2870-43c0-4885-8208-6b0b9998e447-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.354298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.354298 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/f27b2870-43c0-4885-8208-6b0b9998e447-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.354448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.354308 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7qw4\" (UniqueName: \"kubernetes.io/projected/f27b2870-43c0-4885-8208-6b0b9998e447-kube-api-access-l7qw4\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.569551 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.569517 6966 generic.go:356] "Generic (PLEG): container finished" podID="34f216a3-cccd-44ee-8e4b-65d7436c4e08" containerID="1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff" exitCode=0 Oct 30 18:53:47.569703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.569561 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" Oct 30 18:53:47.569703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.569596 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" event={"ID":"34f216a3-cccd-44ee-8e4b-65d7436c4e08","Type":"ContainerDied","Data":"1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff"} Oct 30 18:53:47.569703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.569645 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg" event={"ID":"34f216a3-cccd-44ee-8e4b-65d7436c4e08","Type":"ContainerDied","Data":"80f43a3d918120b150bbcfce6cb4667a41b2a1884101383a0156706c38e423e3"} Oct 30 18:53:47.569703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.569668 6966 scope.go:122] "RemoveContainer" containerID="1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff" Oct 30 18:53:47.571416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.571388 6966 generic.go:356] "Generic (PLEG): container finished" podID="6429ea9c-b300-4a44-897c-fa800c6d8ccc" containerID="d7ee918e84dfec2637fc436a9ff713ac96708c976c063705a9903ed663f82cf3" exitCode=0 Oct 30 18:53:47.571554 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.571504 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"6429ea9c-b300-4a44-897c-fa800c6d8ccc","Type":"ContainerDied","Data":"d7ee918e84dfec2637fc436a9ff713ac96708c976c063705a9903ed663f82cf3"} Oct 30 18:53:47.573738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.573713 6966 generic.go:356] "Generic (PLEG): container finished" podID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerID="e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d" exitCode=0 Oct 30 18:53:47.573843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.573791 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" event={"ID":"fe0d3020-1746-42eb-b9f0-ff9127444f18","Type":"ContainerDied","Data":"e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d"} Oct 30 18:53:47.575329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.575307 6966 generic.go:356] "Generic (PLEG): container finished" podID="f27b2870-43c0-4885-8208-6b0b9998e447" containerID="db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9" exitCode=0 Oct 30 18:53:47.575428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.575359 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" Oct 30 18:53:47.575428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.575391 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" event={"ID":"f27b2870-43c0-4885-8208-6b0b9998e447","Type":"ContainerDied","Data":"db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9"} Oct 30 18:53:47.575540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.575441 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx" event={"ID":"f27b2870-43c0-4885-8208-6b0b9998e447","Type":"ContainerDied","Data":"7db911813698cefb97d34a565e45ac5a7881bff37e05f10cd043e120e8dd33e3"} Oct 30 18:53:47.579816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.579801 6966 scope.go:122] "RemoveContainer" containerID="1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff" Oct 30 18:53:47.580106 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:53:47.580084 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff\": container with ID starting with 1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff not found: ID does not exist" containerID="1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff" Oct 30 18:53:47.580207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.580117 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff"} err="failed to get container status \"1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff\": rpc error: code = NotFound desc = could not find container \"1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff\": container with ID starting with 1dfcecfc874d95d739aecc50f76aeb2e52951342e907db0c6bab3ea7f53bafff not found: ID does not exist" Oct 30 18:53:47.580207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.580139 6966 scope.go:122] "RemoveContainer" containerID="db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9" Oct 30 18:53:47.588408 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.588395 6966 scope.go:122] "RemoveContainer" containerID="db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9" Oct 30 18:53:47.588685 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:53:47.588663 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9\": container with ID starting with db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9 not found: ID does not exist" containerID="db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9" Oct 30 18:53:47.588777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.588693 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9"} err="failed to get container status \"db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9\": rpc error: code = NotFound desc = could not find container \"db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9\": container with ID starting with db120e7a49a4d53b51260f717afbaf584006d2ecd6b05ee870471e9c8078b1a9 not found: ID does not exist" Oct 30 18:53:47.589654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.589628 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:53:47.617299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.617252 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg"] Oct 30 18:53:47.622181 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.622163 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-query-frontend-7d747cfbbdpdzg"] Oct 30 18:53:47.653420 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.653401 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx"] Oct 30 18:53:47.656312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656285 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656332 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj\" (UniqueName: \"kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656385 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656491 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656538 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656656 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656629 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656709 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656662 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret\") pod \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\" (UID: \"6429ea9c-b300-4a44-897c-fa800c6d8ccc\") " Oct 30 18:53:47.656892 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.656839 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.657338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.657311 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:47.658671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.658647 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "kube-api-access-lxbdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:47.658671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.658665 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-collector-74fd5665b8-gr2mx"] Oct 30 18:53:47.658891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.658862 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.658997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.658902 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "data-science-usage-logs-lokistack-compactor-grpc". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.659322 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.659289 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "test-loki-lifecycle-secret". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:47.659858 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.659834 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879" pod "6429ea9c-b300-4a44-897c-fa800c6d8ccc" (UID: "6429ea9c-b300-4a44-897c-fa800c6d8ccc"). InnerVolumeSpecName "pvc-787026af-147e-408d-99c9-3de7d485501a". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:53:47.757969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.757947 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-http\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.757969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.757969 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.758345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.757980 6966 reconciler_common.go:299] "Volume detached for volume \"test-loki-lifecycle-secret\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-test-loki-lifecycle-secret\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.758345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.757990 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-compactor-grpc\" (UniqueName: \"kubernetes.io/secret/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-compactor-grpc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.758345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.758000 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxbdj\" (UniqueName: \"kubernetes.io/projected/6429ea9c-b300-4a44-897c-fa800c6d8ccc-kube-api-access-lxbdj\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.758345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.758009 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6429ea9c-b300-4a44-897c-fa800c6d8ccc-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:47.758345 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.758037 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:53:47.766708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.766693 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-787026af-147e-408d-99c9-3de7d485501a" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:53:47.858855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:47.858836 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-787026af-147e-408d-99c9-3de7d485501a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-051dea544a2323879\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-908cbe7987bf0ef7acb0c43ba5742dca3eade156d34ceb83ea4b9dc800a9dca5\"" Oct 30 18:53:48.091722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.091698 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:53:48.161559 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161529 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161584 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161664 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161715 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161757 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161792 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161849 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.161925 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161891 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret\") pod \"fe0d3020-1746-42eb-b9f0-ff9127444f18\" (UID: \"fe0d3020-1746-42eb-b9f0-ff9127444f18\") " Oct 30 18:53:48.162158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.161941 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "data-science-usage-logs-lokistack-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.162225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.162164 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.164220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.164147 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-ca-bundle". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.164432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.164263 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "data-science-usage-logs-lokistack-gateway-client-http". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.164634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.164594 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "tls-secret". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.164634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.164616 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "kube-api-access-g847q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:53:48.167065 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.167039 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "rbac". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.168595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.168572 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "lokistack-gateway". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:53:48.170312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.170292 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants" pod "fe0d3020-1746-42eb-b9f0-ff9127444f18" (UID: "fe0d3020-1746-42eb-b9f0-ff9127444f18"). InnerVolumeSpecName "tenants". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:53:48.263403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263351 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g847q\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-kube-api-access-g847q\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.263403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263374 6966 reconciler_common.go:299] "Volume detached for volume \"lokistack-gateway\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-lokistack-gateway\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.263403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263383 6966 reconciler_common.go:299] "Volume detached for volume \"rbac\" (UniqueName: \"kubernetes.io/configmap/fe0d3020-1746-42eb-b9f0-ff9127444f18-rbac\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.263403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263393 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-client-http\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-client-http\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.263403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263402 6966 reconciler_common.go:299] "Volume detached for volume \"tenants\" (UniqueName: \"kubernetes.io/secret/fe0d3020-1746-42eb-b9f0-ff9127444f18-tenants\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.263701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263410 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-usage-logs-lokistack-gateway-ca-bundle\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-data-science-usage-logs-lokistack-gateway-ca-bundle\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.263701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.263420 6966 reconciler_common.go:299] "Volume detached for volume \"tls-secret\" (UniqueName: \"kubernetes.io/projected/fe0d3020-1746-42eb-b9f0-ff9127444f18-tls-secret\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:53:48.582721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.582697 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" Oct 30 18:53:48.582976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.582696 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0" event={"ID":"6429ea9c-b300-4a44-897c-fa800c6d8ccc","Type":"ContainerDied","Data":"86701ae9a33cda4f7a44cd1766b726147e50fcc9e4b8376ea20bd8de90367203"} Oct 30 18:53:48.582976 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.582812 6966 scope.go:122] "RemoveContainer" containerID="d7ee918e84dfec2637fc436a9ff713ac96708c976c063705a9903ed663f82cf3" Oct 30 18:53:48.584295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.584270 6966 generic.go:356] "Generic (PLEG): container finished" podID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerID="ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b" exitCode=0 Oct 30 18:53:48.584394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.584309 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" Oct 30 18:53:48.584394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.584318 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" event={"ID":"fe0d3020-1746-42eb-b9f0-ff9127444f18","Type":"ContainerDied","Data":"ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b"} Oct 30 18:53:48.584394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.584348 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v" event={"ID":"fe0d3020-1746-42eb-b9f0-ff9127444f18","Type":"ContainerDied","Data":"df8e7e7425c2d1fffa9c814175a2f1351f5a511e9b94e642249f4ea3ebebf243"} Oct 30 18:53:48.592036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.592021 6966 scope.go:122] "RemoveContainer" containerID="e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d" Oct 30 18:53:48.599872 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.599855 6966 scope.go:122] "RemoveContainer" containerID="ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b" Oct 30 18:53:48.607517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.607498 6966 scope.go:122] "RemoveContainer" containerID="e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d" Oct 30 18:53:48.607763 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:53:48.607746 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d\": container with ID starting with e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d not found: ID does not exist" containerID="e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d" Oct 30 18:53:48.607817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.607771 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d"} err="failed to get container status \"e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d\": rpc error: code = NotFound desc = could not find container \"e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d\": container with ID starting with e339d89bcc989e7a2ad87ce9ed8ceb51a5e5c1c45850236fdd9711ddc26a354d not found: ID does not exist" Oct 30 18:53:48.607817 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.607786 6966 scope.go:122] "RemoveContainer" containerID="ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b" Oct 30 18:53:48.607990 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:53:48.607974 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b\": container with ID starting with ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b not found: ID does not exist" containerID="ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b" Oct 30 18:53:48.608038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.607993 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b"} err="failed to get container status \"ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b\": rpc error: code = NotFound desc = could not find container \"ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b\": container with ID starting with ecfdbc510fbe0c9cf570f0c81bd3aafbc1b950ca3846756c46f7938a362e128b not found: ID does not exist" Oct 30 18:53:48.632148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.632128 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v"] Oct 30 18:53:48.643143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.643120 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-gateway-c5dc6c6c5-vkf4v"] Oct 30 18:53:48.669363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.669341 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:53:48.673961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:48.673943 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-usage-logs-lokistack-compactor-0"] Oct 30 18:53:49.410664 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:49.410634 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f216a3-cccd-44ee-8e4b-65d7436c4e08" path="/var/lib/kubelet/pods/34f216a3-cccd-44ee-8e4b-65d7436c4e08/volumes" Oct 30 18:53:49.411213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:49.411192 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6429ea9c-b300-4a44-897c-fa800c6d8ccc" path="/var/lib/kubelet/pods/6429ea9c-b300-4a44-897c-fa800c6d8ccc/volumes" Oct 30 18:53:49.411619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:49.411602 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f27b2870-43c0-4885-8208-6b0b9998e447" path="/var/lib/kubelet/pods/f27b2870-43c0-4885-8208-6b0b9998e447/volumes" Oct 30 18:53:49.412270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:53:49.412251 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" path="/var/lib/kubelet/pods/fe0d3020-1746-42eb-b9f0-ff9127444f18/volumes" Oct 30 18:54:14.391915 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:14.391872 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:17.198934 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:54:17.198899 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:54:21.369224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.369176 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:21.591275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.591239 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:21.640625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640555 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx"] Oct 30 18:54:21.640905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640892 6966 cpu_manager.go:414] "containerMap: removing container" podUID="34f216a3-cccd-44ee-8e4b-65d7436c4e08" containerName="loki-query-frontend" Oct 30 18:54:21.640947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640908 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f216a3-cccd-44ee-8e4b-65d7436c4e08" containerName="loki-query-frontend" Oct 30 18:54:21.640947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640926 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="opa" Oct 30 18:54:21.640947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640931 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="opa" Oct 30 18:54:21.641049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640950 6966 cpu_manager.go:414] "containerMap: removing container" podUID="f27b2870-43c0-4885-8208-6b0b9998e447" containerName="otc-container" Oct 30 18:54:21.641049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640955 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f27b2870-43c0-4885-8208-6b0b9998e447" containerName="otc-container" Oct 30 18:54:21.641049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640970 6966 cpu_manager.go:414] "containerMap: removing container" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="gateway" Oct 30 18:54:21.641049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640976 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="gateway" Oct 30 18:54:21.641049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640989 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6429ea9c-b300-4a44-897c-fa800c6d8ccc" containerName="loki-compactor" Oct 30 18:54:21.641049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.640997 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6429ea9c-b300-4a44-897c-fa800c6d8ccc" containerName="loki-compactor" Oct 30 18:54:21.641303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.641056 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6429ea9c-b300-4a44-897c-fa800c6d8ccc" containerName="loki-compactor" Oct 30 18:54:21.641303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.641067 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f216a3-cccd-44ee-8e4b-65d7436c4e08" containerName="loki-query-frontend" Oct 30 18:54:21.641303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.641076 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="opa" Oct 30 18:54:21.641303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.641087 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="f27b2870-43c0-4885-8208-6b0b9998e447" containerName="otc-container" Oct 30 18:54:21.641303 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.641095 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0d3020-1746-42eb-b9f0-ff9127444f18" containerName="gateway" Oct 30 18:54:21.644550 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.644531 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.700392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.700367 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx"] Oct 30 18:54:21.722089 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.722069 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.722213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.722117 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppg6l\" (UniqueName: \"kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.722213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.722136 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.730549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.730527 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd"] Oct 30 18:54:21.732359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.732339 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:54:21.734209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.734192 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.739655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.739636 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:54:21.739880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.739863 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:54:21.739979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.739889 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:54:21.739979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.739957 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-mq8xc\"" Oct 30 18:54:21.758574 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.758554 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd"] Oct 30 18:54:21.823263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823239 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.823383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823273 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vkv2\" (UniqueName: \"kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.823383 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823307 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.823560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823543 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.823633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823571 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppg6l\" (UniqueName: \"kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.823633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823592 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.823995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.823972 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.825970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.825950 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.832447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.832427 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:54:21.832595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.832578 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-7r54h\"" Oct 30 18:54:21.832669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.832625 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:54:21.845903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.845886 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppg6l\" (UniqueName: \"kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l\") pod \"data-science-thanos-querier-proxy-6998ff8b98-bk2mx\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:21.869646 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.869612 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:21.924236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.924178 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.924236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.924228 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.924378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.924263 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vkv2\" (UniqueName: \"kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.925057 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.925040 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.926398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.926375 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.946783 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.946759 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vkv2\" (UniqueName: \"kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2\") pod \"korrel8r-5cbcc89b8f-rj6nd\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:21.960601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:21.960584 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:54:22.047302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.047274 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:22.074693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.074662 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.257208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.257140 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx"] Oct 30 18:54:22.261577 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:54:22.261551 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70682acb_3801_4726_a7d5_06948818f7ff.slice/crio-4a18c7d10270dfa7efa8ff5023c3e969a948a645e061af2c679eeab6d7686742 WatchSource:0}: Error finding container 4a18c7d10270dfa7efa8ff5023c3e969a948a645e061af2c679eeab6d7686742: Status 404 returned error can't find the container with id 4a18c7d10270dfa7efa8ff5023c3e969a948a645e061af2c679eeab6d7686742 Oct 30 18:54:22.333613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.333574 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.344644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.344606 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd"] Oct 30 18:54:22.348985 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:54:22.348927 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27023fb9_4c9a_49e9_9422_1844a025fda3.slice/crio-e9b75b61a8e7f557bb3020d530ed34f99c54b5afd041bc94d40e966758bfb2ad WatchSource:0}: Error finding container e9b75b61a8e7f557bb3020d530ed34f99c54b5afd041bc94d40e966758bfb2ad: Status 404 returned error can't find the container with id e9b75b61a8e7f557bb3020d530ed34f99c54b5afd041bc94d40e966758bfb2ad Oct 30 18:54:22.539216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.539148 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.747814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.747785 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" event={"ID":"27023fb9-4c9a-49e9-9422-1844a025fda3","Type":"ContainerStarted","Data":"88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9"} Oct 30 18:54:22.748005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.747837 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" event={"ID":"27023fb9-4c9a-49e9-9422-1844a025fda3","Type":"ContainerStarted","Data":"e9b75b61a8e7f557bb3020d530ed34f99c54b5afd041bc94d40e966758bfb2ad"} Oct 30 18:54:22.748005 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.747867 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:22.750149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.750126 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerStarted","Data":"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6"} Oct 30 18:54:22.750248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.750173 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerStarted","Data":"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844"} Oct 30 18:54:22.750248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.750203 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerStarted","Data":"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f"} Oct 30 18:54:22.750248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.750237 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerStarted","Data":"4a18c7d10270dfa7efa8ff5023c3e969a948a645e061af2c679eeab6d7686742"} Oct 30 18:54:22.753054 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.753031 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:22.789023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.788976 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" podStartSLOduration=1.7889642110000001 podStartE2EDuration="1.788964211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:22.78816096 +0000 UTC m=+3425.494346317" watchObservedRunningTime="2026-10-30 18:54:22.788964211 +0000 UTC m=+3425.495149567" Oct 30 18:54:22.828846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:22.828807 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" podStartSLOduration=1.828793531 podStartE2EDuration="1.828793531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:22.82780586 +0000 UTC m=+3425.533991216" watchObservedRunningTime="2026-10-30 18:54:22.828793531 +0000 UTC m=+3425.534978889" Oct 30 18:54:23.192453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.192371 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:54:23.196933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.196910 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.217214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.217190 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:54:23.227597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.227532 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:54:23.283814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.283781 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:54:23.283971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.283860 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:54:23.283971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.283927 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:54:23.283971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.283952 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:54:23.283971 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.283975 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:54:23.335721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.335681 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.335721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.335716 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.335939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.335862 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hc57\" (UniqueName: \"kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.336002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.335935 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.336002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.335943 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:23.336002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.335975 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.336125 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.336051 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.437092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.437065 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.437272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.437108 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.437272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.437231 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.437393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.437287 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.437393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.437310 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.437393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.437364 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hc57\" (UniqueName: \"kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.438142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.438111 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.439834 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.439807 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.439985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.439861 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.439985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.439945 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.440093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.440017 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.460351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.460298 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hc57\" (UniqueName: \"kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57\") pod \"data-science-collector-collector-1\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.511033 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.511010 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:54:23.554265 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.554227 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:23.811143 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.811118 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:54:23.816412 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:54:23.816383 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f92c7f5_64d8_4872_83a5_2ccff9447816.slice/crio-ddda213a1e31f1f4796115301744f1c79e0fea6d546a6ee851d15407fc70f535 WatchSource:0}: Error finding container ddda213a1e31f1f4796115301744f1c79e0fea6d546a6ee851d15407fc70f535: Status 404 returned error can't find the container with id ddda213a1e31f1f4796115301744f1c79e0fea6d546a6ee851d15407fc70f535 Oct 30 18:54:23.818536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.818505 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:54:23.818627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:23.818563 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:54:24.522373 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:24.522338 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:24.763064 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:24.763027 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f92c7f5-64d8-4872-83a5-2ccff9447816","Type":"ContainerStarted","Data":"d36ad8ada2b01d843545adff89dfd06c2530dfb7dc8127c4efd2ccc39723d6d0"} Oct 30 18:54:24.763445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:24.763078 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f92c7f5-64d8-4872-83a5-2ccff9447816","Type":"ContainerStarted","Data":"ddda213a1e31f1f4796115301744f1c79e0fea6d546a6ee851d15407fc70f535"} Oct 30 18:54:24.805888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:24.805811 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.805799282 podStartE2EDuration="1.805799282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:24.805105456 +0000 UTC m=+3427.511290816" watchObservedRunningTime="2026-10-30 18:54:24.805799282 +0000 UTC m=+3427.511984639" Oct 30 18:54:30.401703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.401612 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.668867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.668790 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.889495 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.889441 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:30.920758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.920687 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:54:30.924556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.924538 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:30.941075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.941050 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:54:30.941419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.941396 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:54:30.941539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.941407 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:54:30.941539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.941436 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:54:30.952298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.952242 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-8dld8\"" Oct 30 18:54:30.956286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:30.956263 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:54:31.010792 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.010762 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:54:31.019048 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.019017 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.035071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.035047 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:54:31.035071 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.035068 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:54:31.035293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.035182 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:54:31.041249 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.041226 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:54:31.042577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.042545 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:54:31.046338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.046314 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:54:31.046338 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.046328 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:54:31.046556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.046313 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-b7kx8\"" Oct 30 18:54:31.046556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.046335 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:54:31.046556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.046357 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:54:31.046556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.046400 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:54:31.106669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106644 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.106669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106675 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.106843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106700 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.106843 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106814 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.106915 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106867 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.106953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106941 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6dmw\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.106988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.106967 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.109883 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.109861 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:31.207592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207527 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.207592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207569 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.207592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207591 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.207795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207607 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.207795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207698 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.207795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207743 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sx4ns\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.207917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207883 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.207917 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207910 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.207992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207942 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.207992 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.207972 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.208087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208018 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.208087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208053 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.208205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208094 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.208205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208123 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.208205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208143 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.208205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208170 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.208205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208202 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.208522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208236 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.208522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208283 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6dmw\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.208522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.208370 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.210246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.210220 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.210377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.210349 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.210453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.210397 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.210544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.210495 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.210544 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.210502 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.230316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.230293 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6dmw\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.243127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.243107 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309074 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309113 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309132 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309148 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sx4ns\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309165 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309186 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309212 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309236 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309260 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309275 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309293 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.309360 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.310454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.310088 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.311281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.310629 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.312328 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.312124 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.313970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.313950 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.316320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.315663 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.316320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.315903 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.316320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.316100 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.316320 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.316158 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.316703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.316388 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.316900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.316859 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.318083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.317822 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.332990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.332967 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sx4ns\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:31.572614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.572575 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:54:31.577176 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:54:31.577149 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7025f2cf_e13c_484f_9da5_56c572a615f3.slice/crio-08e8617542c00a12882e5b6652edde6a23b5b095e1c16df0ef2f27f798f4c25d WatchSource:0}: Error finding container 08e8617542c00a12882e5b6652edde6a23b5b095e1c16df0ef2f27f798f4c25d: Status 404 returned error can't find the container with id 08e8617542c00a12882e5b6652edde6a23b5b095e1c16df0ef2f27f798f4c25d Oct 30 18:54:31.802430 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:31.802399 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerStarted","Data":"08e8617542c00a12882e5b6652edde6a23b5b095e1c16df0ef2f27f798f4c25d"} Oct 30 18:54:32.808908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:32.808832 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerStarted","Data":"f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67"} Oct 30 18:54:33.757274 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:33.757246 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:54:34.540018 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:34.539986 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:54:35.826355 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:35.826327 6966 generic.go:356] "Generic (PLEG): container finished" podID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerID="f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67" exitCode=0 Oct 30 18:54:35.826733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:35.826360 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerDied","Data":"f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67"} Oct 30 18:54:36.838174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:36.838139 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerStarted","Data":"db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7"} Oct 30 18:54:36.838580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:36.838225 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerStarted","Data":"4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72"} Oct 30 18:54:36.838580 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:36.838411 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:36.906897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:36.906848 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.422413708 podStartE2EDuration="6.906837076s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.484423368s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:36.89437786 +0000 UTC m=+3439.600563238" watchObservedRunningTime="2026-10-30 18:54:36.906837076 +0000 UTC m=+3439.613022433" Oct 30 18:54:38.684135 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.684099 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:38.711019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.710990 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:38.785268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.785240 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:38.785391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.785329 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:38.798694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.798667 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:38.897421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.897392 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:38.922738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:38.922717 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:39.141169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:39.141144 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:39.473811 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:39.473737 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:54:39.477484 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:54:39.477437 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbff41649_82e0_491b_90d9_284817ef3437.slice/crio-ed165b8a90be2f71eef0e2bb4279bdaec7bbe77d84f75d85444e08b23aed810d WatchSource:0}: Error finding container ed165b8a90be2f71eef0e2bb4279bdaec7bbe77d84f75d85444e08b23aed810d: Status 404 returned error can't find the container with id ed165b8a90be2f71eef0e2bb4279bdaec7bbe77d84f75d85444e08b23aed810d Oct 30 18:54:39.854401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:39.854366 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerStarted","Data":"ed165b8a90be2f71eef0e2bb4279bdaec7bbe77d84f75d85444e08b23aed810d"} Oct 30 18:54:40.860965 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:40.860934 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerStarted","Data":"4141b86ab18e921f85fa7959a340841c9fba79f24d77f7510f95be19cc680ecb"} Oct 30 18:54:42.843627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:42.843597 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:54:43.880264 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:43.880230 6966 generic.go:356] "Generic (PLEG): container finished" podID="bff41649-82e0-491b-90d9-284817ef3437" containerID="4141b86ab18e921f85fa7959a340841c9fba79f24d77f7510f95be19cc680ecb" exitCode=0 Oct 30 18:54:43.880682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:43.880303 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerDied","Data":"4141b86ab18e921f85fa7959a340841c9fba79f24d77f7510f95be19cc680ecb"} Oct 30 18:54:44.888305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:44.888271 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerStarted","Data":"dc59d329c67e1acf95dff741996f62871e793f5734c7809fbeb9d78b6665bd37"} Oct 30 18:54:44.888696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:44.888332 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerStarted","Data":"779fbfbbf3fb5b34fc97c353cd75410fecf5e72875dfe7711cb98a0816f55ab9"} Oct 30 18:54:44.888696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:44.888357 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerStarted","Data":"81f7cd6953bedb1869c3980f6536238904647498879aea175c27e0d8b5192416"} Oct 30 18:54:44.961847 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:44.961799 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=11.676554948 podStartE2EDuration="14.961787348s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.2852324s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:54:44.952831464 +0000 UTC m=+3447.659016846" watchObservedRunningTime="2026-10-30 18:54:44.961787348 +0000 UTC m=+3447.667972704" Oct 30 18:54:47.199708 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:54:47.199671 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:54:49.141616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:49.141574 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:54.141831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:54.141803 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:54.147063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:54.147041 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:54:54.945763 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:54:54.945738 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:55:06.256478 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:06.256434 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:06.496429 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:06.496390 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:07.860614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:07.860565 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:07.861059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:07.860822 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="3f92c7f5-64d8-4872-83a5-2ccff9447816" containerName="otc-container" containerID="cri-o://d36ad8ada2b01d843545adff89dfd06c2530dfb7dc8127c4efd2ccc39723d6d0" gracePeriod=30 Oct 30 18:55:08.103895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:08.103852 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:08.365503 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:08.365448 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:09.019198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.019166 6966 generic.go:356] "Generic (PLEG): container finished" podID="3f92c7f5-64d8-4872-83a5-2ccff9447816" containerID="d36ad8ada2b01d843545adff89dfd06c2530dfb7dc8127c4efd2ccc39723d6d0" exitCode=0 Oct 30 18:55:09.019579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.019226 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f92c7f5-64d8-4872-83a5-2ccff9447816","Type":"ContainerDied","Data":"d36ad8ada2b01d843545adff89dfd06c2530dfb7dc8127c4efd2ccc39723d6d0"} Oct 30 18:55:09.117170 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.117140 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:09.140352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140324 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs\") pod \"3f92c7f5-64d8-4872-83a5-2ccff9447816\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " Oct 30 18:55:09.140523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140366 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs\") pod \"3f92c7f5-64d8-4872-83a5-2ccff9447816\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " Oct 30 18:55:09.140523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140417 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57\" (UniqueName: \"kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57\") pod \"3f92c7f5-64d8-4872-83a5-2ccff9447816\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " Oct 30 18:55:09.140523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140491 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs\") pod \"3f92c7f5-64d8-4872-83a5-2ccff9447816\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " Oct 30 18:55:09.140523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140521 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal\") pod \"3f92c7f5-64d8-4872-83a5-2ccff9447816\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " Oct 30 18:55:09.141119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140872 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal" pod "3f92c7f5-64d8-4872-83a5-2ccff9447816" (UID: "3f92c7f5-64d8-4872-83a5-2ccff9447816"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:09.141119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.140877 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert\") pod \"3f92c7f5-64d8-4872-83a5-2ccff9447816\" (UID: \"3f92c7f5-64d8-4872-83a5-2ccff9447816\") " Oct 30 18:55:09.141288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.141235 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/3f92c7f5-64d8-4872-83a5-2ccff9447816-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:09.144601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.144578 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs" pod "3f92c7f5-64d8-4872-83a5-2ccff9447816" (UID: "3f92c7f5-64d8-4872-83a5-2ccff9447816"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:09.144748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.144591 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs" pod "3f92c7f5-64d8-4872-83a5-2ccff9447816" (UID: "3f92c7f5-64d8-4872-83a5-2ccff9447816"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:09.144748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.144602 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57" pod "3f92c7f5-64d8-4872-83a5-2ccff9447816" (UID: "3f92c7f5-64d8-4872-83a5-2ccff9447816"). InnerVolumeSpecName "kube-api-access-2hc57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:09.144748 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.144639 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs" pod "3f92c7f5-64d8-4872-83a5-2ccff9447816" (UID: "3f92c7f5-64d8-4872-83a5-2ccff9447816"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:09.147838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.147816 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert" pod "3f92c7f5-64d8-4872-83a5-2ccff9447816" (UID: "3f92c7f5-64d8-4872-83a5-2ccff9447816"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:09.242418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.242325 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:09.242418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.242361 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:09.242418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.242375 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:09.242418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.242388 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f92c7f5-64d8-4872-83a5-2ccff9447816-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:09.242418 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:09.242400 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hc57\" (UniqueName: \"kubernetes.io/projected/3f92c7f5-64d8-4872-83a5-2ccff9447816-kube-api-access-2hc57\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:10.026037 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.026003 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.026557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.026009 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"3f92c7f5-64d8-4872-83a5-2ccff9447816","Type":"ContainerDied","Data":"ddda213a1e31f1f4796115301744f1c79e0fea6d546a6ee851d15407fc70f535"} Oct 30 18:55:10.026557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.026134 6966 scope.go:122] "RemoveContainer" containerID="d36ad8ada2b01d843545adff89dfd06c2530dfb7dc8127c4efd2ccc39723d6d0" Oct 30 18:55:10.075318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.075296 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:10.086099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.086077 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:10.118152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.118131 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:10.118527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.118506 6966 cpu_manager.go:414] "containerMap: removing container" podUID="3f92c7f5-64d8-4872-83a5-2ccff9447816" containerName="otc-container" Oct 30 18:55:10.118527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.118528 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f92c7f5-64d8-4872-83a5-2ccff9447816" containerName="otc-container" Oct 30 18:55:10.118734 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.118631 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f92c7f5-64d8-4872-83a5-2ccff9447816" containerName="otc-container" Oct 30 18:55:10.124098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.124069 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.138860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.138833 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:55:10.138860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.138853 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:55:10.139036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.138868 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:55:10.139036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.138913 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:55:10.139036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.138991 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:55:10.139036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.139031 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:55:10.143837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.143819 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:10.150834 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.150807 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.150978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.150956 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l6tbx\" (UniqueName: \"kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.151053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.151004 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.151113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.151063 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.151113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.151102 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.151225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.151160 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.252186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.252162 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.252311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.252194 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.252443 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.252418 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l6tbx\" (UniqueName: \"kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.252516 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.252495 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.252567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.252550 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.252621 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.252592 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.253305 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.253279 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.254802 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.254763 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.254902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.254872 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.255392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.255370 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.255450 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.255420 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.274639 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.274618 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l6tbx\" (UniqueName: \"kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx\") pod \"data-science-collector-collector-1\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.443095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.443065 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:10.764780 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.764648 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:10.769219 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:55:10.769187 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023cbce0_3575_439b_a2f3_8cada8b6f908.slice/crio-2b78034fd96c41a76a79abf34b050fdb04f0536699589dced338997254a6ee97 WatchSource:0}: Error finding container 2b78034fd96c41a76a79abf34b050fdb04f0536699589dced338997254a6ee97: Status 404 returned error can't find the container with id 2b78034fd96c41a76a79abf34b050fdb04f0536699589dced338997254a6ee97 Oct 30 18:55:10.771723 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.771690 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:55:10.771815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:10.771748 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:55:11.035579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:11.035497 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"023cbce0-3575-439b-a2f3-8cada8b6f908","Type":"ContainerStarted","Data":"889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be"} Oct 30 18:55:11.035579 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:11.035551 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"023cbce0-3575-439b-a2f3-8cada8b6f908","Type":"ContainerStarted","Data":"2b78034fd96c41a76a79abf34b050fdb04f0536699589dced338997254a6ee97"} Oct 30 18:55:11.079182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:11.079106 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.079091679 podStartE2EDuration="1.079091679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:11.076294427 +0000 UTC m=+3473.782479785" watchObservedRunningTime="2026-10-30 18:55:11.079091679 +0000 UTC m=+3473.785277030" Oct 30 18:55:11.411757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:11.411723 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f92c7f5-64d8-4872-83a5-2ccff9447816" path="/var/lib/kubelet/pods/3f92c7f5-64d8-4872-83a5-2ccff9447816/volumes" Oct 30 18:55:17.199041 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:55:17.199003 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:55:23.660502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:23.660443 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:23.927962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:23.927881 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:24.137474 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:24.137393 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:24.649714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:24.649658 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:25.326195 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.326165 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:25.327016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.326381 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="023cbce0-3575-439b-a2f3-8cada8b6f908" containerName="otc-container" containerID="cri-o://889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be" gracePeriod=30 Oct 30 18:55:25.557868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.557825 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:25.583780 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.583724 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:25.687865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.687832 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal\" (UniqueName: \"kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal\") pod \"023cbce0-3575-439b-a2f3-8cada8b6f908\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " Oct 30 18:55:25.688020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.687882 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs\") pod \"023cbce0-3575-439b-a2f3-8cada8b6f908\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " Oct 30 18:55:25.688020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.687914 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs\") pod \"023cbce0-3575-439b-a2f3-8cada8b6f908\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " Oct 30 18:55:25.688020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.687975 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs\") pod \"023cbce0-3575-439b-a2f3-8cada8b6f908\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " Oct 30 18:55:25.688020 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.687999 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert\") pod \"023cbce0-3575-439b-a2f3-8cada8b6f908\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " Oct 30 18:55:25.688225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.688032 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx\" (UniqueName: \"kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx\") pod \"023cbce0-3575-439b-a2f3-8cada8b6f908\" (UID: \"023cbce0-3575-439b-a2f3-8cada8b6f908\") " Oct 30 18:55:25.688292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.688267 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal" pod "023cbce0-3575-439b-a2f3-8cada8b6f908" (UID: "023cbce0-3575-439b-a2f3-8cada8b6f908"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:55:25.688385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.688367 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/023cbce0-3575-439b-a2f3-8cada8b6f908-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:25.690169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.690139 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs" pod "023cbce0-3575-439b-a2f3-8cada8b6f908" (UID: "023cbce0-3575-439b-a2f3-8cada8b6f908"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:25.690628 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.690595 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs" pod "023cbce0-3575-439b-a2f3-8cada8b6f908" (UID: "023cbce0-3575-439b-a2f3-8cada8b6f908"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:25.690735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.690620 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx" pod "023cbce0-3575-439b-a2f3-8cada8b6f908" (UID: "023cbce0-3575-439b-a2f3-8cada8b6f908"). InnerVolumeSpecName "kube-api-access-l6tbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:55:25.690735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.690663 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs" pod "023cbce0-3575-439b-a2f3-8cada8b6f908" (UID: "023cbce0-3575-439b-a2f3-8cada8b6f908"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:25.690735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.690690 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert" pod "023cbce0-3575-439b-a2f3-8cada8b6f908" (UID: "023cbce0-3575-439b-a2f3-8cada8b6f908"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:55:25.768060 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.768007 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:25.789252 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.789223 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:25.789252 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.789251 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:25.789414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.789265 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:25.789414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.789279 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/023cbce0-3575-439b-a2f3-8cada8b6f908-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:25.789414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:25.789291 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l6tbx\" (UniqueName: \"kubernetes.io/projected/023cbce0-3575-439b-a2f3-8cada8b6f908-kube-api-access-l6tbx\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:55:26.112164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.112127 6966 generic.go:356] "Generic (PLEG): container finished" podID="023cbce0-3575-439b-a2f3-8cada8b6f908" containerID="889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be" exitCode=0 Oct 30 18:55:26.112331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.112180 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.112331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.112210 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"023cbce0-3575-439b-a2f3-8cada8b6f908","Type":"ContainerDied","Data":"889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be"} Oct 30 18:55:26.112331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.112266 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"023cbce0-3575-439b-a2f3-8cada8b6f908","Type":"ContainerDied","Data":"2b78034fd96c41a76a79abf34b050fdb04f0536699589dced338997254a6ee97"} Oct 30 18:55:26.112331 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.112288 6966 scope.go:122] "RemoveContainer" containerID="889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be" Oct 30 18:55:26.121620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.121602 6966 scope.go:122] "RemoveContainer" containerID="889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be" Oct 30 18:55:26.121884 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:55:26.121868 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be\": container with ID starting with 889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be not found: ID does not exist" containerID="889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be" Oct 30 18:55:26.121937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.121892 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be"} err="failed to get container status \"889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be\": rpc error: code = NotFound desc = could not find container \"889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be\": container with ID starting with 889a47046c633802405b95ba01d915ba4ca2045b7b9f615c3454634e383d50be not found: ID does not exist" Oct 30 18:55:26.160269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.160215 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:26.172587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.172563 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:26.206254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.206229 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:26.206875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.206854 6966 cpu_manager.go:414] "containerMap: removing container" podUID="023cbce0-3575-439b-a2f3-8cada8b6f908" containerName="otc-container" Oct 30 18:55:26.206875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.206872 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="023cbce0-3575-439b-a2f3-8cada8b6f908" containerName="otc-container" Oct 30 18:55:26.207026 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.206966 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="023cbce0-3575-439b-a2f3-8cada8b6f908" containerName="otc-container" Oct 30 18:55:26.212157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.212138 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.226893 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.226873 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:55:26.226893 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.226872 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:55:26.229975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.229955 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:55:26.230088 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.230006 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:55:26.231730 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.231706 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:26.293122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293094 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:55:26.293247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293159 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.293247 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293200 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.293352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293332 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrbgb\" (UniqueName: \"kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.293396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293383 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.293449 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293433 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.293553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.293535 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.327283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.327259 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:55:26.394129 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394068 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrbgb\" (UniqueName: \"kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.394129 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394105 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.394129 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394126 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.394337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394163 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.394337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394215 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.394337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394239 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.394888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.394864 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.396785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.396755 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.396904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.396854 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.396904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.396858 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.397078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.397055 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.415939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.415919 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrbgb\" (UniqueName: \"kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb\") pod \"data-science-collector-collector-1\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.529806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.529780 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:55:26.830914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.830885 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:55:26.836509 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:55:26.836451 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c63e020_8953_44fa_89af_0b14919ab2c4.slice/crio-d8642b42664eb7f1f45b01177107ec0542dc09d5de2e6b0401c1d29446456c1d WatchSource:0}: Error finding container d8642b42664eb7f1f45b01177107ec0542dc09d5de2e6b0401c1d29446456c1d: Status 404 returned error can't find the container with id d8642b42664eb7f1f45b01177107ec0542dc09d5de2e6b0401c1d29446456c1d Oct 30 18:55:26.838886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.838845 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:55:26.838995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:26.838910 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:55:27.120312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:27.120186 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c63e020-8953-44fa-89af-0b14919ab2c4","Type":"ContainerStarted","Data":"c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59"} Oct 30 18:55:27.120312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:27.120282 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c63e020-8953-44fa-89af-0b14919ab2c4","Type":"ContainerStarted","Data":"d8642b42664eb7f1f45b01177107ec0542dc09d5de2e6b0401c1d29446456c1d"} Oct 30 18:55:27.162904 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:27.162854 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.162841698 podStartE2EDuration="1.162841698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:55:27.161198267 +0000 UTC m=+3489.867383621" watchObservedRunningTime="2026-10-30 18:55:27.162841698 +0000 UTC m=+3489.869027055" Oct 30 18:55:27.411601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:27.411536 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="023cbce0-3575-439b-a2f3-8cada8b6f908" path="/var/lib/kubelet/pods/023cbce0-3575-439b-a2f3-8cada8b6f908/volumes" Oct 30 18:55:47.199340 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:55:47.199305 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:55:58.829057 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:58.828969 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:59.053912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:59.053860 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:55:59.259676 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:55:59.259589 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:00.463948 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:00.463908 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:00.464486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:00.464129 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="5c63e020-8953-44fa-89af-0b14919ab2c4" containerName="otc-container" containerID="cri-o://c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59" gracePeriod=30 Oct 30 18:56:00.699389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:00.699343 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:00.921780 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:00.921747 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:02.808482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.808438 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:02.893304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893239 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs\") pod \"5c63e020-8953-44fa-89af-0b14919ab2c4\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " Oct 30 18:56:02.893304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893272 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal\") pod \"5c63e020-8953-44fa-89af-0b14919ab2c4\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " Oct 30 18:56:02.893304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893295 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert\") pod \"5c63e020-8953-44fa-89af-0b14919ab2c4\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " Oct 30 18:56:02.893575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893341 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb\" (UniqueName: \"kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb\") pod \"5c63e020-8953-44fa-89af-0b14919ab2c4\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " Oct 30 18:56:02.893575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893361 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs\") pod \"5c63e020-8953-44fa-89af-0b14919ab2c4\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " Oct 30 18:56:02.893575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893414 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs\") pod \"5c63e020-8953-44fa-89af-0b14919ab2c4\" (UID: \"5c63e020-8953-44fa-89af-0b14919ab2c4\") " Oct 30 18:56:02.893753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.893702 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal" pod "5c63e020-8953-44fa-89af-0b14919ab2c4" (UID: "5c63e020-8953-44fa-89af-0b14919ab2c4"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:02.895562 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.895527 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs" pod "5c63e020-8953-44fa-89af-0b14919ab2c4" (UID: "5c63e020-8953-44fa-89af-0b14919ab2c4"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:02.895687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.895630 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs" pod "5c63e020-8953-44fa-89af-0b14919ab2c4" (UID: "5c63e020-8953-44fa-89af-0b14919ab2c4"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:02.895738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.895714 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert" pod "5c63e020-8953-44fa-89af-0b14919ab2c4" (UID: "5c63e020-8953-44fa-89af-0b14919ab2c4"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:02.896013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.895992 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs" pod "5c63e020-8953-44fa-89af-0b14919ab2c4" (UID: "5c63e020-8953-44fa-89af-0b14919ab2c4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:02.896058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.896016 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb" pod "5c63e020-8953-44fa-89af-0b14919ab2c4" (UID: "5c63e020-8953-44fa-89af-0b14919ab2c4"). InnerVolumeSpecName "kube-api-access-wrbgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:02.994975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.994949 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrbgb\" (UniqueName: \"kubernetes.io/projected/5c63e020-8953-44fa-89af-0b14919ab2c4-kube-api-access-wrbgb\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:02.994975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.994972 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:02.995152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.994983 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:02.995152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.994991 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:02.995152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.994999 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/5c63e020-8953-44fa-89af-0b14919ab2c4-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:02.995152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:02.995009 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/5c63e020-8953-44fa-89af-0b14919ab2c4-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:03.310741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.310668 6966 generic.go:356] "Generic (PLEG): container finished" podID="5c63e020-8953-44fa-89af-0b14919ab2c4" containerID="c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59" exitCode=0 Oct 30 18:56:03.310741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.310714 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.310741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.310725 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c63e020-8953-44fa-89af-0b14919ab2c4","Type":"ContainerDied","Data":"c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59"} Oct 30 18:56:03.310957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.310792 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"5c63e020-8953-44fa-89af-0b14919ab2c4","Type":"ContainerDied","Data":"d8642b42664eb7f1f45b01177107ec0542dc09d5de2e6b0401c1d29446456c1d"} Oct 30 18:56:03.310957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.310813 6966 scope.go:122] "RemoveContainer" containerID="c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59" Oct 30 18:56:03.320175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.320157 6966 scope.go:122] "RemoveContainer" containerID="c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59" Oct 30 18:56:03.320422 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:03.320404 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59\": container with ID starting with c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59 not found: ID does not exist" containerID="c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59" Oct 30 18:56:03.320540 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.320430 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59"} err="failed to get container status \"c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59\": rpc error: code = NotFound desc = could not find container \"c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59\": container with ID starting with c9d34780f4ddec8eab04348698cd8262fe5ec0a210b63870c1c5a331e789eb59 not found: ID does not exist" Oct 30 18:56:03.359557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.359535 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:03.370571 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.370551 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:03.401121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.401097 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:03.401501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.401480 6966 cpu_manager.go:414] "containerMap: removing container" podUID="5c63e020-8953-44fa-89af-0b14919ab2c4" containerName="otc-container" Oct 30 18:56:03.401501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.401497 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c63e020-8953-44fa-89af-0b14919ab2c4" containerName="otc-container" Oct 30 18:56:03.401714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.401692 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c63e020-8953-44fa-89af-0b14919ab2c4" containerName="otc-container" Oct 30 18:56:03.404734 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.404715 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.411402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.411375 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c63e020-8953-44fa-89af-0b14919ab2c4" path="/var/lib/kubelet/pods/5c63e020-8953-44fa-89af-0b14919ab2c4/volumes" Oct 30 18:56:03.418658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.418639 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-cnw9r\"" Oct 30 18:56:03.418762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.418667 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:56:03.418762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.418714 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:56:03.418762 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.418762 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:56:03.418887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.418805 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:56:03.418887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.418855 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:56:03.431513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.431492 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:03.499768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.499743 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.499908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.499772 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.499908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.499793 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.500036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.499966 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h52s8\" (UniqueName: \"kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.500097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.500037 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.500097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.500085 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601071 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601103 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601155 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h52s8\" (UniqueName: \"kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601203 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601241 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601296 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.601994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.601974 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.603776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.603757 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.603863 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.603797 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.603905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.603872 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.603905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.603890 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.623552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.623535 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h52s8\" (UniqueName: \"kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8\") pod \"data-science-collector-collector-1\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:03.717583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:03.717554 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:04.021509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:04.021408 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:04.026013 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:04.025983 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e26c298_d29c_4e48_a3e9_44c9ce15c582.slice/crio-185cebf0be002671e86616a0649dbc3bc7e48a55b56112b09f93e62f57b5a894 WatchSource:0}: Error finding container 185cebf0be002671e86616a0649dbc3bc7e48a55b56112b09f93e62f57b5a894: Status 404 returned error can't find the container with id 185cebf0be002671e86616a0649dbc3bc7e48a55b56112b09f93e62f57b5a894 Oct 30 18:56:04.028010 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:04.027980 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:56:04.028102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:04.028044 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:56:04.317583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:04.317543 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8e26c298-d29c-4e48-a3e9-44c9ce15c582","Type":"ContainerStarted","Data":"47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a"} Oct 30 18:56:04.317764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:04.317609 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8e26c298-d29c-4e48-a3e9-44c9ce15c582","Type":"ContainerStarted","Data":"185cebf0be002671e86616a0649dbc3bc7e48a55b56112b09f93e62f57b5a894"} Oct 30 18:56:04.359714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:04.359666 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.359654508 podStartE2EDuration="1.359654508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:04.359114846 +0000 UTC m=+3527.065300203" watchObservedRunningTime="2026-10-30 18:56:04.359654508 +0000 UTC m=+3527.065839864" Oct 30 18:56:09.308380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.308341 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx"] Oct 30 18:56:09.309271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.308685 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="kube-rbac-proxy" containerID="cri-o://484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" gracePeriod=30 Oct 30 18:56:09.309271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.308723 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="prom-label-proxy" containerID="cri-o://6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" gracePeriod=30 Oct 30 18:56:09.309271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.308934 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="method-gate" containerID="cri-o://7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" gracePeriod=30 Oct 30 18:56:09.327281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.327259 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd"] Oct 30 18:56:09.327513 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.327441 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" podUID="27023fb9-4c9a-49e9-9422-1844a025fda3" containerName="korrel8r" containerID="cri-o://88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9" gracePeriod=30 Oct 30 18:56:09.590313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.590289 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:56:09.603833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.603808 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:56:09.655326 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.655298 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l\" (UniqueName: \"kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l\") pod \"70682acb-3801-4726-a7d5-06948818f7ff\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " Oct 30 18:56:09.655536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.655343 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls\" (UniqueName: \"kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls\") pod \"70682acb-3801-4726-a7d5-06948818f7ff\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " Oct 30 18:56:09.655536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.655370 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config\" (UniqueName: \"kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config\") pod \"27023fb9-4c9a-49e9-9422-1844a025fda3\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " Oct 30 18:56:09.655536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.655397 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config\" (UniqueName: \"kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config\") pod \"70682acb-3801-4726-a7d5-06948818f7ff\" (UID: \"70682acb-3801-4726-a7d5-06948818f7ff\") " Oct 30 18:56:09.655536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.655428 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls\" (UniqueName: \"kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls\") pod \"27023fb9-4c9a-49e9-9422-1844a025fda3\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " Oct 30 18:56:09.655536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.655452 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2\" (UniqueName: \"kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2\") pod \"27023fb9-4c9a-49e9-9422-1844a025fda3\" (UID: \"27023fb9-4c9a-49e9-9422-1844a025fda3\") " Oct 30 18:56:09.657541 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.657497 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config" pod "70682acb-3801-4726-a7d5-06948818f7ff" (UID: "70682acb-3801-4726-a7d5-06948818f7ff"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:09.658470 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.658359 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config" pod "27023fb9-4c9a-49e9-9422-1844a025fda3" (UID: "27023fb9-4c9a-49e9-9422-1844a025fda3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:09.658912 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.658891 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l" pod "70682acb-3801-4726-a7d5-06948818f7ff" (UID: "70682acb-3801-4726-a7d5-06948818f7ff"). InnerVolumeSpecName "kube-api-access-ppg6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:09.659041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.658928 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2" pod "27023fb9-4c9a-49e9-9422-1844a025fda3" (UID: "27023fb9-4c9a-49e9-9422-1844a025fda3"). InnerVolumeSpecName "kube-api-access-7vkv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:09.659552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.659523 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls" pod "27023fb9-4c9a-49e9-9422-1844a025fda3" (UID: "27023fb9-4c9a-49e9-9422-1844a025fda3"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:09.659808 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.659736 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls" pod "70682acb-3801-4726-a7d5-06948818f7ff" (UID: "70682acb-3801-4726-a7d5-06948818f7ff"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:09.756704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.756673 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppg6l\" (UniqueName: \"kubernetes.io/projected/70682acb-3801-4726-a7d5-06948818f7ff-kube-api-access-ppg6l\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.756704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.756697 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/70682acb-3801-4726-a7d5-06948818f7ff-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.756704 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.756707 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/27023fb9-4c9a-49e9-9422-1844a025fda3-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.756939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.756716 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/70682acb-3801-4726-a7d5-06948818f7ff-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.756939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.756724 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/27023fb9-4c9a-49e9-9422-1844a025fda3-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.756939 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.756732 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vkv2\" (UniqueName: \"kubernetes.io/projected/27023fb9-4c9a-49e9-9422-1844a025fda3-kube-api-access-7vkv2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:09.774836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.774804 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:56:09.775134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.775110 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="prometheus" containerID="cri-o://81f7cd6953bedb1869c3980f6536238904647498879aea175c27e0d8b5192416" gracePeriod=600 Oct 30 18:56:09.775525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.775502 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="thanos-sidecar" containerID="cri-o://dc59d329c67e1acf95dff741996f62871e793f5734c7809fbeb9d78b6665bd37" gracePeriod=600 Oct 30 18:56:09.775666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.775647 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="config-reloader" containerID="cri-o://779fbfbbf3fb5b34fc97c353cd75410fecf5e72875dfe7711cb98a0816f55ab9" gracePeriod=600 Oct 30 18:56:09.786371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.786312 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:56:09.786652 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.786620 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="alertmanager" containerID="cri-o://4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72" gracePeriod=120 Oct 30 18:56:09.786768 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.786678 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="config-reloader" containerID="cri-o://db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7" gracePeriod=120 Oct 30 18:56:09.817087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.817034 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:09.941957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:09.941919 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="prometheus" probeResult="failure" output="Get \"https://10.134.0.143:9090/-/ready\": dial tcp 10.134.0.143:9090: connect: connection refused" Oct 30 18:56:10.087523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.087437 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:10.292092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.292053 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:10.361400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.361342 6966 generic.go:356] "Generic (PLEG): container finished" podID="bff41649-82e0-491b-90d9-284817ef3437" containerID="dc59d329c67e1acf95dff741996f62871e793f5734c7809fbeb9d78b6665bd37" exitCode=0 Oct 30 18:56:10.361400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.361362 6966 generic.go:356] "Generic (PLEG): container finished" podID="bff41649-82e0-491b-90d9-284817ef3437" containerID="779fbfbbf3fb5b34fc97c353cd75410fecf5e72875dfe7711cb98a0816f55ab9" exitCode=0 Oct 30 18:56:10.361400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.361369 6966 generic.go:356] "Generic (PLEG): container finished" podID="bff41649-82e0-491b-90d9-284817ef3437" containerID="81f7cd6953bedb1869c3980f6536238904647498879aea175c27e0d8b5192416" exitCode=0 Oct 30 18:56:10.361897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.361416 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerDied","Data":"dc59d329c67e1acf95dff741996f62871e793f5734c7809fbeb9d78b6665bd37"} Oct 30 18:56:10.361897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.361514 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerDied","Data":"779fbfbbf3fb5b34fc97c353cd75410fecf5e72875dfe7711cb98a0816f55ab9"} Oct 30 18:56:10.361897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.361559 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerDied","Data":"81f7cd6953bedb1869c3980f6536238904647498879aea175c27e0d8b5192416"} Oct 30 18:56:10.363256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363233 6966 generic.go:356] "Generic (PLEG): container finished" podID="70682acb-3801-4726-a7d5-06948818f7ff" containerID="6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" exitCode=0 Oct 30 18:56:10.363256 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363256 6966 generic.go:356] "Generic (PLEG): container finished" podID="70682acb-3801-4726-a7d5-06948818f7ff" containerID="7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" exitCode=0 Oct 30 18:56:10.363403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363264 6966 generic.go:356] "Generic (PLEG): container finished" podID="70682acb-3801-4726-a7d5-06948818f7ff" containerID="484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" exitCode=0 Oct 30 18:56:10.363403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363290 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" Oct 30 18:56:10.363403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363310 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerDied","Data":"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6"} Oct 30 18:56:10.363403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363364 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerDied","Data":"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844"} Oct 30 18:56:10.363403 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363391 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerDied","Data":"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f"} Oct 30 18:56:10.363672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363422 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx" event={"ID":"70682acb-3801-4726-a7d5-06948818f7ff","Type":"ContainerDied","Data":"4a18c7d10270dfa7efa8ff5023c3e969a948a645e061af2c679eeab6d7686742"} Oct 30 18:56:10.363672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.363482 6966 scope.go:122] "RemoveContainer" containerID="6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" Oct 30 18:56:10.369363 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.369340 6966 generic.go:356] "Generic (PLEG): container finished" podID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerID="db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7" exitCode=0 Oct 30 18:56:10.369494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.369446 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerDied","Data":"db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7"} Oct 30 18:56:10.370861 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.370842 6966 generic.go:356] "Generic (PLEG): container finished" podID="27023fb9-4c9a-49e9-9422-1844a025fda3" containerID="88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9" exitCode=0 Oct 30 18:56:10.370936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.370874 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" event={"ID":"27023fb9-4c9a-49e9-9422-1844a025fda3","Type":"ContainerDied","Data":"88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9"} Oct 30 18:56:10.370936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.370891 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" Oct 30 18:56:10.370936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.370917 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd" event={"ID":"27023fb9-4c9a-49e9-9422-1844a025fda3","Type":"ContainerDied","Data":"e9b75b61a8e7f557bb3020d530ed34f99c54b5afd041bc94d40e966758bfb2ad"} Oct 30 18:56:10.382818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.379412 6966 scope.go:122] "RemoveContainer" containerID="7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" Oct 30 18:56:10.393123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.392518 6966 scope.go:122] "RemoveContainer" containerID="484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" Oct 30 18:56:10.403619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.403593 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:10.403827 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.403801 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="8e26c298-d29c-4e48-a3e9-44c9ce15c582" containerName="otc-container" containerID="cri-o://47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a" gracePeriod=30 Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.405355 6966 scope.go:122] "RemoveContainer" containerID="6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:10.405692 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": container with ID starting with 6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6 not found: ID does not exist" containerID="6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.405731 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6"} err="failed to get container status \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": rpc error: code = NotFound desc = could not find container \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": container with ID starting with 6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6 not found: ID does not exist" Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.405756 6966 scope.go:122] "RemoveContainer" containerID="7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:10.406068 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": container with ID starting with 7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844 not found: ID does not exist" containerID="7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.406102 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844"} err="failed to get container status \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": rpc error: code = NotFound desc = could not find container \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": container with ID starting with 7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844 not found: ID does not exist" Oct 30 18:56:10.406285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.406123 6966 scope.go:122] "RemoveContainer" containerID="484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" Oct 30 18:56:10.407121 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:10.406392 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": container with ID starting with 484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f not found: ID does not exist" containerID="484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" Oct 30 18:56:10.407121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.406420 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f"} err="failed to get container status \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": rpc error: code = NotFound desc = could not find container \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": container with ID starting with 484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f not found: ID does not exist" Oct 30 18:56:10.407121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.406439 6966 scope.go:122] "RemoveContainer" containerID="6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" Oct 30 18:56:10.407121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407015 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6"} err="failed to get container status \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": rpc error: code = NotFound desc = could not find container \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": container with ID starting with 6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6 not found: ID does not exist" Oct 30 18:56:10.407121 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407037 6966 scope.go:122] "RemoveContainer" containerID="7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" Oct 30 18:56:10.407421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407272 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844"} err="failed to get container status \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": rpc error: code = NotFound desc = could not find container \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": container with ID starting with 7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844 not found: ID does not exist" Oct 30 18:56:10.407421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407295 6966 scope.go:122] "RemoveContainer" containerID="484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" Oct 30 18:56:10.407631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407575 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f"} err="failed to get container status \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": rpc error: code = NotFound desc = could not find container \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": container with ID starting with 484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f not found: ID does not exist" Oct 30 18:56:10.407631 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407600 6966 scope.go:122] "RemoveContainer" containerID="6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6" Oct 30 18:56:10.407942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407903 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6"} err="failed to get container status \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": rpc error: code = NotFound desc = could not find container \"6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6\": container with ID starting with 6ba3ff6a4aff476a9b3b3ae5342813e4f7d2bb414c5216481e1fb8d9fdfc05d6 not found: ID does not exist" Oct 30 18:56:10.407942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.407930 6966 scope.go:122] "RemoveContainer" containerID="7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844" Oct 30 18:56:10.408309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.408195 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844"} err="failed to get container status \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": rpc error: code = NotFound desc = could not find container \"7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844\": container with ID starting with 7809d54229a6d9ce3fc35f66023fee6f5edc76a809101134d8c079ed71458844 not found: ID does not exist" Oct 30 18:56:10.408309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.408222 6966 scope.go:122] "RemoveContainer" containerID="484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f" Oct 30 18:56:10.408520 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.408493 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f"} err="failed to get container status \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": rpc error: code = NotFound desc = could not find container \"484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f\": container with ID starting with 484864cd3922254efb42419aea5d1e877f83f6b090d1cfd8390725cd23dac33f not found: ID does not exist" Oct 30 18:56:10.408622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.408520 6966 scope.go:122] "RemoveContainer" containerID="88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9" Oct 30 18:56:10.419484 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.417026 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx"] Oct 30 18:56:10.428224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.428206 6966 scope.go:122] "RemoveContainer" containerID="88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9" Oct 30 18:56:10.429067 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:10.429010 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9\": container with ID starting with 88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9 not found: ID does not exist" containerID="88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9" Oct 30 18:56:10.429213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.429069 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9"} err="failed to get container status \"88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9\": rpc error: code = NotFound desc = could not find container \"88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9\": container with ID starting with 88121583ea79f9d367ffb2c139dfb3f343150bdb6628638bbe93ea70dfbe42d9 not found: ID does not exist" Oct 30 18:56:10.430282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.430260 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-bk2mx"] Oct 30 18:56:10.511852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.511785 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd"] Oct 30 18:56:10.520127 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.520101 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-rj6nd"] Oct 30 18:56:10.536472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.536432 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:10.608479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.608444 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:10.648813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.648789 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:10.662017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.661995 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662027 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662052 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662079 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662156 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out\" (UniqueName: \"kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662174 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662217 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662278 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662250 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662288 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662318 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662356 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662519 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662474 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662532 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"bff41649-82e0-491b-90d9-284817ef3437\" (UID: \"bff41649-82e0-491b-90d9-284817ef3437\") " Oct 30 18:56:10.662741 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662595 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.662905 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.662853 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.663427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.663098 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.664058 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.664029 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.664708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.664680 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.665252 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.665218 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "kube-api-access-sx4ns". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.665340 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.665323 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.665942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.665903 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:56:10.666292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.666255 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.666820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.666788 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.666820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.666810 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.668203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.668171 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.672744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.672723 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:56:10.679154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.679133 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config" pod "bff41649-82e0-491b-90d9-284817ef3437" (UID: "bff41649-82e0-491b-90d9-284817ef3437"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.754967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.754936 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:10.763750 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.763725 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs\") pod \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " Oct 30 18:56:10.763881 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.763770 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs\") pod \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " Oct 30 18:56:10.763881 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.763804 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8\" (UniqueName: \"kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8\") pod \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " Oct 30 18:56:10.763985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.763902 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal\" (UniqueName: \"kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal\") pod \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " Oct 30 18:56:10.763985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.763962 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert\") pod \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " Oct 30 18:56:10.764110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764001 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs\") pod \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\" (UID: \"8e26c298-d29c-4e48-a3e9-44c9ce15c582\") " Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764230 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764254 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764272 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764287 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/bff41649-82e0-491b-90d9-284817ef3437-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764302 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764316 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764329 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764342 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sx4ns\" (UniqueName: \"kubernetes.io/projected/bff41649-82e0-491b-90d9-284817ef3437-kube-api-access-sx4ns\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764357 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/bff41649-82e0-491b-90d9-284817ef3437-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764372 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764392 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal" pod "8e26c298-d29c-4e48-a3e9-44c9ce15c582" (UID: "8e26c298-d29c-4e48-a3e9-44c9ce15c582"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764402 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:56:10.766092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.764489 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/bff41649-82e0-491b-90d9-284817ef3437-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.766823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.766592 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs" pod "8e26c298-d29c-4e48-a3e9-44c9ce15c582" (UID: "8e26c298-d29c-4e48-a3e9-44c9ce15c582"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.766823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.766619 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert" pod "8e26c298-d29c-4e48-a3e9-44c9ce15c582" (UID: "8e26c298-d29c-4e48-a3e9-44c9ce15c582"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.766823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.766738 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8" pod "8e26c298-d29c-4e48-a3e9-44c9ce15c582" (UID: "8e26c298-d29c-4e48-a3e9-44c9ce15c582"). InnerVolumeSpecName "kube-api-access-h52s8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:10.766962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.766938 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs" pod "8e26c298-d29c-4e48-a3e9-44c9ce15c582" (UID: "8e26c298-d29c-4e48-a3e9-44c9ce15c582"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.767277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.767258 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs" pod "8e26c298-d29c-4e48-a3e9-44c9ce15c582" (UID: "8e26c298-d29c-4e48-a3e9-44c9ce15c582"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:10.777708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.777692 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:56:10.865309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865287 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.865309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865310 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.865491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865322 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 18:56:10.865491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865332 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.865491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865340 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.865491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865348 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/8e26c298-d29c-4e48-a3e9-44c9ce15c582-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:10.865491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:10.865355 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h52s8\" (UniqueName: \"kubernetes.io/projected/8e26c298-d29c-4e48-a3e9-44c9ce15c582-kube-api-access-h52s8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:11.379618 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.379585 6966 generic.go:356] "Generic (PLEG): container finished" podID="8e26c298-d29c-4e48-a3e9-44c9ce15c582" containerID="47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a" exitCode=0 Oct 30 18:56:11.380086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.379647 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:11.380086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.379670 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8e26c298-d29c-4e48-a3e9-44c9ce15c582","Type":"ContainerDied","Data":"47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a"} Oct 30 18:56:11.380086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.379769 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"8e26c298-d29c-4e48-a3e9-44c9ce15c582","Type":"ContainerDied","Data":"185cebf0be002671e86616a0649dbc3bc7e48a55b56112b09f93e62f57b5a894"} Oct 30 18:56:11.380086 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.379806 6966 scope.go:122] "RemoveContainer" containerID="47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a" Oct 30 18:56:11.383243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.383222 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"bff41649-82e0-491b-90d9-284817ef3437","Type":"ContainerDied","Data":"ed165b8a90be2f71eef0e2bb4279bdaec7bbe77d84f75d85444e08b23aed810d"} Oct 30 18:56:11.383397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.383380 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:11.402288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.397935 6966 scope.go:122] "RemoveContainer" containerID="47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a" Oct 30 18:56:11.402288 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:11.398916 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a\": container with ID starting with 47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a not found: ID does not exist" containerID="47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a" Oct 30 18:56:11.402288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.398941 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a"} err="failed to get container status \"47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a\": rpc error: code = NotFound desc = could not find container \"47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a\": container with ID starting with 47f1447ddce1732136fb7fb22763778996d6bf234e9de512f7d30780fe0e7b9a not found: ID does not exist" Oct 30 18:56:11.402288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.398956 6966 scope.go:122] "RemoveContainer" containerID="dc59d329c67e1acf95dff741996f62871e793f5734c7809fbeb9d78b6665bd37" Oct 30 18:56:11.407594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.407446 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:11.409727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.409573 6966 scope.go:122] "RemoveContainer" containerID="779fbfbbf3fb5b34fc97c353cd75410fecf5e72875dfe7711cb98a0816f55ab9" Oct 30 18:56:11.413167 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.413144 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27023fb9-4c9a-49e9-9422-1844a025fda3" path="/var/lib/kubelet/pods/27023fb9-4c9a-49e9-9422-1844a025fda3/volumes" Oct 30 18:56:11.413776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.413753 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70682acb-3801-4726-a7d5-06948818f7ff" path="/var/lib/kubelet/pods/70682acb-3801-4726-a7d5-06948818f7ff/volumes" Oct 30 18:56:11.420186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.420165 6966 scope.go:122] "RemoveContainer" containerID="81f7cd6953bedb1869c3980f6536238904647498879aea175c27e0d8b5192416" Oct 30 18:56:11.432009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.431988 6966 scope.go:122] "RemoveContainer" containerID="4141b86ab18e921f85fa7959a340841c9fba79f24d77f7510f95be19cc680ecb" Oct 30 18:56:11.447785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.447758 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:56:11.453341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.453290 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:56:11.484267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.484229 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:11.488530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.488511 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:11.628594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:11.628524 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:12.843824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:12.843785 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.142:9093/-/ready\": dial tcp 10.134.0.142:9093: connect: connection refused" Oct 30 18:56:13.411511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:13.411454 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e26c298-d29c-4e48-a3e9-44c9ce15c582" path="/var/lib/kubelet/pods/8e26c298-d29c-4e48-a3e9-44c9ce15c582/volumes" Oct 30 18:56:13.411936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:13.411923 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bff41649-82e0-491b-90d9-284817ef3437" path="/var/lib/kubelet/pods/bff41649-82e0-491b-90d9-284817ef3437/volumes" Oct 30 18:56:13.789972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:13.789873 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:17.199332 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:17.199286 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:56:17.842984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:17.842949 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.142:9093/-/ready\": dial tcp 10.134.0.142:9093: connect: connection refused" Oct 30 18:56:20.027325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.027302 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:20.156241 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156217 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156267 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156291 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156331 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156352 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156392 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156419 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out\") pod \"7025f2cf-e13c-484f-9da5-56c572a615f3\" (UID: \"7025f2cf-e13c-484f-9da5-56c572a615f3\") " Oct 30 18:56:20.156865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.156830 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:56:20.158836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.158807 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:20.158959 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.158813 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "kube-api-access-c6dmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:20.159047 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.159015 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:56:20.159102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.159043 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:20.162447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.162426 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:20.170805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.170783 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config" pod "7025f2cf-e13c-484f-9da5-56c572a615f3" (UID: "7025f2cf-e13c-484f-9da5-56c572a615f3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:20.257386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257353 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.257386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257388 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6dmw\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-kube-api-access-c6dmw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.257614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257403 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.257614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257417 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7025f2cf-e13c-484f-9da5-56c572a615f3-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.257614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257430 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7025f2cf-e13c-484f-9da5-56c572a615f3-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.257614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257444 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.257614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.257486 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7025f2cf-e13c-484f-9da5-56c572a615f3-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:20.342133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.342096 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:20.445561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.445490 6966 generic.go:356] "Generic (PLEG): container finished" podID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerID="4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72" exitCode=0 Oct 30 18:56:20.445561 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.445548 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:20.445805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.445584 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerDied","Data":"4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72"} Oct 30 18:56:20.445805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.445662 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"7025f2cf-e13c-484f-9da5-56c572a615f3","Type":"ContainerDied","Data":"08e8617542c00a12882e5b6652edde6a23b5b095e1c16df0ef2f27f798f4c25d"} Oct 30 18:56:20.445805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.445702 6966 scope.go:122] "RemoveContainer" containerID="db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7" Oct 30 18:56:20.455175 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.455057 6966 scope.go:122] "RemoveContainer" containerID="4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72" Oct 30 18:56:20.463404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.463379 6966 scope.go:122] "RemoveContainer" containerID="f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67" Oct 30 18:56:20.472343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.472321 6966 scope.go:122] "RemoveContainer" containerID="db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7" Oct 30 18:56:20.472645 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:20.472621 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7\": container with ID starting with db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7 not found: ID does not exist" containerID="db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7" Oct 30 18:56:20.472722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.472657 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7"} err="failed to get container status \"db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7\": rpc error: code = NotFound desc = could not find container \"db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7\": container with ID starting with db5e18514b63ec6aa70202fcafcf45047436c0074c7f0170db42e6002e5ba6f7 not found: ID does not exist" Oct 30 18:56:20.472722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.472678 6966 scope.go:122] "RemoveContainer" containerID="4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72" Oct 30 18:56:20.472962 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:20.472940 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72\": container with ID starting with 4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72 not found: ID does not exist" containerID="4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72" Oct 30 18:56:20.473023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.472972 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72"} err="failed to get container status \"4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72\": rpc error: code = NotFound desc = could not find container \"4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72\": container with ID starting with 4e03fcb3226ff5b6f9c13dc1275452aca9cd8e721778bb3b2f90d5c01c8b9f72 not found: ID does not exist" Oct 30 18:56:20.473023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.472996 6966 scope.go:122] "RemoveContainer" containerID="f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67" Oct 30 18:56:20.473257 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:20.473238 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67\": container with ID starting with f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67 not found: ID does not exist" containerID="f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67" Oct 30 18:56:20.473317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.473265 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67"} err="failed to get container status \"f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67\": rpc error: code = NotFound desc = could not find container \"f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67\": container with ID starting with f08934ec805f090e1154e3f09c0767270e49fa41e8f8609e753cb8e4631afd67 not found: ID does not exist" Oct 30 18:56:20.496787 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.496767 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:56:20.503528 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.503508 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:56:20.552889 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.552865 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:20.620998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.620972 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h"] Oct 30 18:56:20.621327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621310 6966 cpu_manager.go:414] "containerMap: removing container" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="method-gate" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621329 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="method-gate" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621342 6966 cpu_manager.go:414] "containerMap: removing container" podUID="27023fb9-4c9a-49e9-9422-1844a025fda3" containerName="korrel8r" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621348 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27023fb9-4c9a-49e9-9422-1844a025fda3" containerName="korrel8r" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621355 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="init-config-reloader" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621360 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="init-config-reloader" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621367 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="alertmanager" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621371 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="alertmanager" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621377 6966 cpu_manager.go:414] "containerMap: removing container" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="config-reloader" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621383 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="config-reloader" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621393 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="config-reloader" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621400 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="config-reloader" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621416 6966 cpu_manager.go:414] "containerMap: removing container" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="kube-rbac-proxy" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621421 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="kube-rbac-proxy" Oct 30 18:56:20.621423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621427 6966 cpu_manager.go:414] "containerMap: removing container" podUID="8e26c298-d29c-4e48-a3e9-44c9ce15c582" containerName="otc-container" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621432 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e26c298-d29c-4e48-a3e9-44c9ce15c582" containerName="otc-container" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621440 6966 cpu_manager.go:414] "containerMap: removing container" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="thanos-sidecar" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621445 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="thanos-sidecar" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621454 6966 cpu_manager.go:414] "containerMap: removing container" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="prom-label-proxy" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621623 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="prom-label-proxy" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621820 6966 cpu_manager.go:414] "containerMap: removing container" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="init-config-reloader" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621830 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="init-config-reloader" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621846 6966 cpu_manager.go:414] "containerMap: removing container" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="prometheus" Oct 30 18:56:20.622160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.621862 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="prometheus" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622223 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="method-gate" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622262 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="27023fb9-4c9a-49e9-9422-1844a025fda3" containerName="korrel8r" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622280 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="config-reloader" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622289 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="alertmanager" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622299 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" containerName="config-reloader" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622317 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="kube-rbac-proxy" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622326 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e26c298-d29c-4e48-a3e9-44c9ce15c582" containerName="otc-container" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622336 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="thanos-sidecar" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622419 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="70682acb-3801-4726-a7d5-06948818f7ff" containerName="prom-label-proxy" Oct 30 18:56:20.622605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.622431 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="bff41649-82e0-491b-90d9-284817ef3437" containerName="prometheus" Oct 30 18:56:20.628635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.628614 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.645894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.645869 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h"] Oct 30 18:56:20.659381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.659362 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:56:20.659491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.659386 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:56:20.659491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.659395 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:56:20.662017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.661999 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6"] Oct 30 18:56:20.662397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.662337 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:56:20.662397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.662347 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-bk692\"" Oct 30 18:56:20.665103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.665086 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.686067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.686049 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6"] Oct 30 18:56:20.727868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.727815 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:56:20.727868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.727832 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-2sqsj\"" Oct 30 18:56:20.760534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.760501 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:56:20.762837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.762819 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkrtw\" (UniqueName: \"kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.762916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.762847 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.762916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.762868 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.762916 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.762891 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.763073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.763013 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8b6s\" (UniqueName: \"kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.763118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.763106 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.780000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.779979 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:20.864198 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.864170 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.864292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.864252 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkrtw\" (UniqueName: \"kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.864292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.864281 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.864414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.864309 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.864414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.864336 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.864414 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.864364 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8b6s\" (UniqueName: \"kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.865023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.865004 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.865132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.865114 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.866616 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.866597 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.866797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.866779 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.886423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.886397 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8b6s\" (UniqueName: \"kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s\") pod \"korrel8r-5cbcc89b8f-799l6\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:20.886660 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.886642 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkrtw\" (UniqueName: \"kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw\") pod \"data-science-thanos-querier-proxy-6998ff8b98-v6v4h\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.945859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.945837 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:56:20.976636 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:20.976611 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:21.231276 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.231195 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.243272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.243247 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h"] Oct 30 18:56:21.252859 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:21.252596 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad2f0836_8daf_41f7_a61f_48fd8012f46b.slice/crio-71e6a80394b2605855c171d7e0cb3241bc91c5505435d1d250c05860983f0594 WatchSource:0}: Error finding container 71e6a80394b2605855c171d7e0cb3241bc91c5505435d1d250c05860983f0594: Status 404 returned error can't find the container with id 71e6a80394b2605855c171d7e0cb3241bc91c5505435d1d250c05860983f0594 Oct 30 18:56:21.283906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.283878 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6"] Oct 30 18:56:21.289353 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:21.289327 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869b5b61_27b3_4883_a591_9398e2e2ad11.slice/crio-27311dc116b811ebcf2c860d2b0f572448ab986faf3e370dfa3adbaeaea71923 WatchSource:0}: Error finding container 27311dc116b811ebcf2c860d2b0f572448ab986faf3e370dfa3adbaeaea71923: Status 404 returned error can't find the container with id 27311dc116b811ebcf2c860d2b0f572448ab986faf3e370dfa3adbaeaea71923 Oct 30 18:56:21.411115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.411091 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7025f2cf-e13c-484f-9da5-56c572a615f3" path="/var/lib/kubelet/pods/7025f2cf-e13c-484f-9da5-56c572a615f3/volumes" Oct 30 18:56:21.436987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.436919 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.451880 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.451846 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" event={"ID":"869b5b61-27b3-4883-a591-9398e2e2ad11","Type":"ContainerStarted","Data":"abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714"} Oct 30 18:56:21.451996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.451894 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" event={"ID":"869b5b61-27b3-4883-a591-9398e2e2ad11","Type":"ContainerStarted","Data":"27311dc116b811ebcf2c860d2b0f572448ab986faf3e370dfa3adbaeaea71923"} Oct 30 18:56:21.451996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.451950 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:21.455012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.454989 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerStarted","Data":"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453"} Oct 30 18:56:21.455136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.455039 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerStarted","Data":"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f"} Oct 30 18:56:21.455136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.455064 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerStarted","Data":"71e6a80394b2605855c171d7e0cb3241bc91c5505435d1d250c05860983f0594"} Oct 30 18:56:21.491027 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.490957 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" podStartSLOduration=1.4909453 podStartE2EDuration="1.4909453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:21.489556784 +0000 UTC m=+3544.195742142" watchObservedRunningTime="2026-10-30 18:56:21.4909453 +0000 UTC m=+3544.197130657" Oct 30 18:56:21.736297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.736259 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:21.939387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:21.939355 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:22.125431 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.125390 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:22.128864 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.128843 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.147634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.147613 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:56:22.157227 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.157206 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:22.206830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.206723 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:56:22.206954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.206835 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:56:22.206954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.206870 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:56:22.206954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.206874 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-p8bgw\"" Oct 30 18:56:22.206954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.206912 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:56:22.210810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.210775 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:22.276400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.276367 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.276810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.276408 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.276810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.276438 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlscv\" (UniqueName: \"kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.276810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.276559 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.276810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.276599 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.276810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.276680 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.378093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.378063 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.378239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.378115 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.378239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.378137 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.378239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.378165 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.378239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.378192 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlscv\" (UniqueName: \"kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.378239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.378217 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.379053 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.379024 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.380691 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.380666 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.380776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.380697 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.380776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.380721 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.380926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.380905 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.402885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.402864 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlscv\" (UniqueName: \"kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv\") pod \"data-science-collector-collector-1\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.423157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.423126 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:22.443093 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.443070 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:22.462171 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.462107 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerStarted","Data":"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2"} Oct 30 18:56:22.503227 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.503169 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" podStartSLOduration=2.5031497529999998 podStartE2EDuration="2.503149753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:22.501815503 +0000 UTC m=+3545.208000864" watchObservedRunningTime="2026-10-30 18:56:22.503149753 +0000 UTC m=+3545.209335111" Oct 30 18:56:22.635407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.635374 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:22.741108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.741004 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:22.745645 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:22.745608 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod184e20eb_87f8_4aba_bc35_b1d3918952b5.slice/crio-5a4c93a7810301891023b6c2454d1305ac7d26afba393e40a01e0c31e29c7e02 WatchSource:0}: Error finding container 5a4c93a7810301891023b6c2454d1305ac7d26afba393e40a01e0c31e29c7e02: Status 404 returned error can't find the container with id 5a4c93a7810301891023b6c2454d1305ac7d26afba393e40a01e0c31e29c7e02 Oct 30 18:56:22.747865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.747833 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:56:22.747956 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:22.747896 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:56:23.469097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:23.469060 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"184e20eb-87f8-4aba-bc35-b1d3918952b5","Type":"ContainerStarted","Data":"9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e"} Oct 30 18:56:23.471744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:23.469121 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"184e20eb-87f8-4aba-bc35-b1d3918952b5","Type":"ContainerStarted","Data":"5a4c93a7810301891023b6c2454d1305ac7d26afba393e40a01e0c31e29c7e02"} Oct 30 18:56:23.511019 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:23.510975 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.510962423 podStartE2EDuration="1.510962423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:23.510301713 +0000 UTC m=+3546.216487072" watchObservedRunningTime="2026-10-30 18:56:23.510962423 +0000 UTC m=+3546.217147779" Oct 30 18:56:23.781930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:23.781826 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:24.779851 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:24.779815 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:29.470985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:29.470946 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:29.721404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:29.721325 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:29.926078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:29.926044 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.031268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.031190 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:56:30.035297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.035273 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.052680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.052639 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:56:30.052680 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.052639 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:56:30.052902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.052864 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:56:30.052902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.052897 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:56:30.063209 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.063186 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-7rf6z\"" Oct 30 18:56:30.065509 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.065484 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:56:30.114657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.114632 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:56:30.118667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.118646 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.134389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.134369 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:56:30.134512 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.134391 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:56:30.134908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.134893 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:56:30.146608 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146587 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.146721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146692 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.146793 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146743 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhdzr\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.146839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146826 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.146896 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146856 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.147009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146888 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.147009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146907 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.147009 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.146994 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.149482 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.149445 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:56:30.152502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.152421 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:56:30.152502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.152423 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:56:30.152502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.152422 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:56:30.152747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.152552 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:56:30.152747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.152678 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-786bx\"" Oct 30 18:56:30.152747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.152706 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:56:30.153163 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.153133 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:56:30.248132 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248104 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248137 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248168 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248245 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.248419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248294 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.248419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248327 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248359 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9v4tf\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248391 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248452 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.248677 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248559 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248706 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248727 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.248796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248753 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.248958 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248870 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhdzr\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.248958 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248921 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.249126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248958 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.249126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.248992 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.249126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.249031 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.249126 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.249062 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.249536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.249502 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.250974 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.250951 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.251083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.250977 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.251083 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.250990 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.251202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.251079 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.251556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.251540 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.271231 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.271207 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhdzr\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.350192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350157 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350199 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350229 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350352 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350383 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350429 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350481 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9v4tf\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350518 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350544 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350581 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350603 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350685 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.350907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.350804 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.351082 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.351006 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.353119 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.353095 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.353387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.353360 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.353525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.353363 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.353606 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.353524 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.353721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.353698 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.354035 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.353978 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:30.354130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.354079 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.354425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.354404 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.354776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.354754 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.354891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.354872 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.374823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.374794 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9v4tf\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:30.445620 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.445587 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:30.689419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:30.689314 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:56:30.694123 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:30.694097 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ab7571_0308_484b_af6c_1e91865d28fa.slice/crio-723462b3f4e911dd0c5774d6deb4f10863a8881114ba5b2582b445481133f5a9 WatchSource:0}: Error finding container 723462b3f4e911dd0c5774d6deb4f10863a8881114ba5b2582b445481133f5a9: Status 404 returned error can't find the container with id 723462b3f4e911dd0c5774d6deb4f10863a8881114ba5b2582b445481133f5a9 Oct 30 18:56:31.510385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:31.510281 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerStarted","Data":"b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69"} Oct 30 18:56:31.510385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:31.510350 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerStarted","Data":"723462b3f4e911dd0c5774d6deb4f10863a8881114ba5b2582b445481133f5a9"} Oct 30 18:56:32.464534 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.464504 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:56:32.574140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.573908 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.600837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.600810 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.675257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.675205 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.675367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.675321 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.688813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.688793 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.787329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.787296 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.812536 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.812513 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:32.839146 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:32.839122 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:33.171323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:33.171296 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:56:33.175606 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:33.175578 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod837885a4_532e_488d_b7a9_a79fcbbe4ae3.slice/crio-81d51098bdbbc9fe965b4a35c045052371a3b0c20b39b803182f9a0ca63bd9d6 WatchSource:0}: Error finding container 81d51098bdbbc9fe965b4a35c045052371a3b0c20b39b803182f9a0ca63bd9d6: Status 404 returned error can't find the container with id 81d51098bdbbc9fe965b4a35c045052371a3b0c20b39b803182f9a0ca63bd9d6 Oct 30 18:56:33.521313 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:33.521239 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerStarted","Data":"81d51098bdbbc9fe965b4a35c045052371a3b0c20b39b803182f9a0ca63bd9d6"} Oct 30 18:56:34.528983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:34.528945 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerStarted","Data":"b9fb37ab93fc27c86e8e38498b9acfc9bf8e84cc65f81735cca6cf4f2664bafd"} Oct 30 18:56:35.535354 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:35.535322 6966 generic.go:356] "Generic (PLEG): container finished" podID="43ab7571-0308-484b-af6c-1e91865d28fa" containerID="b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69" exitCode=0 Oct 30 18:56:35.535737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:35.535393 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerDied","Data":"b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69"} Oct 30 18:56:36.543242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:36.543210 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerStarted","Data":"922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b"} Oct 30 18:56:36.543764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:36.543280 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerStarted","Data":"30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a"} Oct 30 18:56:36.543764 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:36.543510 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:36.613602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:36.613558 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.028977904 podStartE2EDuration="6.613545604s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.5845677s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:36.612840118 +0000 UTC m=+3559.319025576" watchObservedRunningTime="2026-10-30 18:56:36.613545604 +0000 UTC m=+3559.319730961" Oct 30 18:56:37.551056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:37.550975 6966 generic.go:356] "Generic (PLEG): container finished" podID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerID="b9fb37ab93fc27c86e8e38498b9acfc9bf8e84cc65f81735cca6cf4f2664bafd" exitCode=0 Oct 30 18:56:37.551056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:37.551043 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerDied","Data":"b9fb37ab93fc27c86e8e38498b9acfc9bf8e84cc65f81735cca6cf4f2664bafd"} Oct 30 18:56:38.559491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:38.559341 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerStarted","Data":"997fe1ddf00150ccb162ee4b2a6ffbb81ec68b6bea4955565ea0b5adbfee7f1f"} Oct 30 18:56:38.559491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:38.559453 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerStarted","Data":"ca6f71fdd1d961cac9bb5aab29bf5edbf66c1de371246ec0f9e4574b3151e8ab"} Oct 30 18:56:38.559994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:38.559516 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerStarted","Data":"d8594b3cdaefeae1717a54ed3ebbe1949188aeced82042d2ba7ba1bd7223ec66"} Oct 30 18:56:38.633483 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:38.633414 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=4.947591344 podStartE2EDuration="8.633401007s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.685809663s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:56:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:38.621275243 +0000 UTC m=+3561.327460601" watchObservedRunningTime="2026-10-30 18:56:38.633401007 +0000 UTC m=+3561.339586363" Oct 30 18:56:42.552255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:42.552228 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:56:42.839306 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:42.839269 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:47.199426 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:47.199391 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:56:47.839714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:47.839682 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:47.844801 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:47.844781 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:48.611150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:48.611124 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:56:51.868003 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:51.867968 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:52.094982 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:52.094948 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:52.301873 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:52.301793 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:53.524107 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.524075 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:53.524604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.524418 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="184e20eb-87f8-4aba-bc35-b1d3918952b5" containerName="otc-container" containerID="cri-o://9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e" gracePeriod=30 Oct 30 18:56:53.764995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.764969 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:53.770308 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.770288 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:53.868506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.868481 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs\") pod \"184e20eb-87f8-4aba-bc35-b1d3918952b5\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " Oct 30 18:56:53.868679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.868552 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert\") pod \"184e20eb-87f8-4aba-bc35-b1d3918952b5\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " Oct 30 18:56:53.868679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.868589 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs\") pod \"184e20eb-87f8-4aba-bc35-b1d3918952b5\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " Oct 30 18:56:53.868679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.868621 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv\" (UniqueName: \"kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv\") pod \"184e20eb-87f8-4aba-bc35-b1d3918952b5\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " Oct 30 18:56:53.868679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.868665 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal\") pod \"184e20eb-87f8-4aba-bc35-b1d3918952b5\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " Oct 30 18:56:53.868902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.868702 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs\") pod \"184e20eb-87f8-4aba-bc35-b1d3918952b5\" (UID: \"184e20eb-87f8-4aba-bc35-b1d3918952b5\") " Oct 30 18:56:53.869098 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.869069 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal" pod "184e20eb-87f8-4aba-bc35-b1d3918952b5" (UID: "184e20eb-87f8-4aba-bc35-b1d3918952b5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:56:53.870951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.870917 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs" pod "184e20eb-87f8-4aba-bc35-b1d3918952b5" (UID: "184e20eb-87f8-4aba-bc35-b1d3918952b5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:53.870951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.870928 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs" pod "184e20eb-87f8-4aba-bc35-b1d3918952b5" (UID: "184e20eb-87f8-4aba-bc35-b1d3918952b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:53.871094 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.870955 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv" pod "184e20eb-87f8-4aba-bc35-b1d3918952b5" (UID: "184e20eb-87f8-4aba-bc35-b1d3918952b5"). InnerVolumeSpecName "kube-api-access-nlscv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:56:53.871145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.871106 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs" pod "184e20eb-87f8-4aba-bc35-b1d3918952b5" (UID: "184e20eb-87f8-4aba-bc35-b1d3918952b5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:53.871224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.871209 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert" pod "184e20eb-87f8-4aba-bc35-b1d3918952b5" (UID: "184e20eb-87f8-4aba-bc35-b1d3918952b5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:56:53.969415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.969382 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:53.969415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.969413 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlscv\" (UniqueName: \"kubernetes.io/projected/184e20eb-87f8-4aba-bc35-b1d3918952b5-kube-api-access-nlscv\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:53.969585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.969429 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/184e20eb-87f8-4aba-bc35-b1d3918952b5-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:53.969585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.969441 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:53.969585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.969453 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:53.969585 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:53.969485 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/184e20eb-87f8-4aba-bc35-b1d3918952b5-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:56:54.037296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.037267 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:56:54.645658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.645613 6966 generic.go:356] "Generic (PLEG): container finished" podID="184e20eb-87f8-4aba-bc35-b1d3918952b5" containerID="9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e" exitCode=0 Oct 30 18:56:54.646044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.645685 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.646044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.645691 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"184e20eb-87f8-4aba-bc35-b1d3918952b5","Type":"ContainerDied","Data":"9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e"} Oct 30 18:56:54.646044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.645765 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"184e20eb-87f8-4aba-bc35-b1d3918952b5","Type":"ContainerDied","Data":"5a4c93a7810301891023b6c2454d1305ac7d26afba393e40a01e0c31e29c7e02"} Oct 30 18:56:54.646044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.645788 6966 scope.go:122] "RemoveContainer" containerID="9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e" Oct 30 18:56:54.655874 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.655858 6966 scope.go:122] "RemoveContainer" containerID="9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e" Oct 30 18:56:54.656146 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:56:54.656128 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e\": container with ID starting with 9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e not found: ID does not exist" containerID="9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e" Oct 30 18:56:54.656207 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.656153 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e"} err="failed to get container status \"9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e\": rpc error: code = NotFound desc = could not find container \"9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e\": container with ID starting with 9bf80051ed43c8f1d767f380eb927cc23bd0e98140113db436b0d1a2e8a0418e not found: ID does not exist" Oct 30 18:56:54.725918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.725895 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:54.742157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.742129 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:54.778165 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.778133 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:54.778735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.778716 6966 cpu_manager.go:414] "containerMap: removing container" podUID="184e20eb-87f8-4aba-bc35-b1d3918952b5" containerName="otc-container" Oct 30 18:56:54.778804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.778739 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="184e20eb-87f8-4aba-bc35-b1d3918952b5" containerName="otc-container" Oct 30 18:56:54.778860 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.778848 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="184e20eb-87f8-4aba-bc35-b1d3918952b5" containerName="otc-container" Oct 30 18:56:54.783270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.783254 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.797689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.797667 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:56:54.797689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.797678 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-p8bgw\"" Oct 30 18:56:54.797833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.797692 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:56:54.797833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.797712 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:56:54.797833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.797727 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:56:54.808761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.808735 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:56:54.810701 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.810651 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:54.877404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.877379 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.877548 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.877419 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.877548 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.877456 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.877711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.877578 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.877711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.877620 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.877711 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.877657 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdczg\" (UniqueName: \"kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.978727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.978668 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.978727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.978706 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.978894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.978731 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.978894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.978767 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.978894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.978826 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.978894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.978859 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdczg\" (UniqueName: \"kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.979632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.979606 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.981497 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.981446 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.981687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.981589 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.981687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.981607 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:54.981687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:54.981611 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:55.002447 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.002422 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdczg\" (UniqueName: \"kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg\") pod \"data-science-collector-collector-1\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:55.097095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.097071 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:56:55.399794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.399763 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:56:55.404772 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:56:55.404741 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddae06927_a3bf_4a96_8da3_99871d1801a5.slice/crio-a9d8cedd7ab2796a422d8587a2bf9f94b88fc66c8b319216dafe0658350037b5 WatchSource:0}: Error finding container a9d8cedd7ab2796a422d8587a2bf9f94b88fc66c8b319216dafe0658350037b5: Status 404 returned error can't find the container with id a9d8cedd7ab2796a422d8587a2bf9f94b88fc66c8b319216dafe0658350037b5 Oct 30 18:56:55.407267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.407233 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:56:55.407365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.407297 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:56:55.413185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.413156 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="184e20eb-87f8-4aba-bc35-b1d3918952b5" path="/var/lib/kubelet/pods/184e20eb-87f8-4aba-bc35-b1d3918952b5/volumes" Oct 30 18:56:55.653779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.653677 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dae06927-a3bf-4a96-8da3-99871d1801a5","Type":"ContainerStarted","Data":"e238a819e4e4335537ac935d1ee9f5156d9c87ddd6599f72b70dc45b37253f7b"} Oct 30 18:56:55.653779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.653736 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dae06927-a3bf-4a96-8da3-99871d1801a5","Type":"ContainerStarted","Data":"a9d8cedd7ab2796a422d8587a2bf9f94b88fc66c8b319216dafe0658350037b5"} Oct 30 18:56:55.695454 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:56:55.695406 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.695393303 podStartE2EDuration="1.695393303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:56:55.694240661 +0000 UTC m=+3578.400426017" watchObservedRunningTime="2026-10-30 18:56:55.695393303 +0000 UTC m=+3578.401578660" Oct 30 18:57:04.115547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:04.115511 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:04.340312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:04.340283 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:08.782987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:08.782953 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:09.084717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:09.084683 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:09.293577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:09.293539 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:10.410910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:10.410878 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:10.411393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:10.411092 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="dae06927-a3bf-4a96-8da3-99871d1801a5" containerName="otc-container" containerID="cri-o://e238a819e4e4335537ac935d1ee9f5156d9c87ddd6599f72b70dc45b37253f7b" gracePeriod=30 Oct 30 18:57:10.646178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:10.646132 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:10.851476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:10.851404 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:12.744044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.744015 6966 generic.go:356] "Generic (PLEG): container finished" podID="dae06927-a3bf-4a96-8da3-99871d1801a5" containerID="e238a819e4e4335537ac935d1ee9f5156d9c87ddd6599f72b70dc45b37253f7b" exitCode=0 Oct 30 18:57:12.744396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.744060 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dae06927-a3bf-4a96-8da3-99871d1801a5","Type":"ContainerDied","Data":"e238a819e4e4335537ac935d1ee9f5156d9c87ddd6599f72b70dc45b37253f7b"} Oct 30 18:57:12.762397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.762374 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:12.825024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.824996 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs\") pod \"dae06927-a3bf-4a96-8da3-99871d1801a5\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " Oct 30 18:57:12.825178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.825043 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert\") pod \"dae06927-a3bf-4a96-8da3-99871d1801a5\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " Oct 30 18:57:12.825178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.825082 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs\") pod \"dae06927-a3bf-4a96-8da3-99871d1801a5\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " Oct 30 18:57:12.825178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.825147 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg\" (UniqueName: \"kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg\") pod \"dae06927-a3bf-4a96-8da3-99871d1801a5\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " Oct 30 18:57:12.825359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.825225 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs\") pod \"dae06927-a3bf-4a96-8da3-99871d1801a5\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " Oct 30 18:57:12.825359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.825259 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal\" (UniqueName: \"kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal\") pod \"dae06927-a3bf-4a96-8da3-99871d1801a5\" (UID: \"dae06927-a3bf-4a96-8da3-99871d1801a5\") " Oct 30 18:57:12.825979 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.825884 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal" pod "dae06927-a3bf-4a96-8da3-99871d1801a5" (UID: "dae06927-a3bf-4a96-8da3-99871d1801a5"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:12.827421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.827396 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs" pod "dae06927-a3bf-4a96-8da3-99871d1801a5" (UID: "dae06927-a3bf-4a96-8da3-99871d1801a5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:12.827671 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.827647 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert" pod "dae06927-a3bf-4a96-8da3-99871d1801a5" (UID: "dae06927-a3bf-4a96-8da3-99871d1801a5"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:12.827786 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.827767 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg" pod "dae06927-a3bf-4a96-8da3-99871d1801a5" (UID: "dae06927-a3bf-4a96-8da3-99871d1801a5"). InnerVolumeSpecName "kube-api-access-hdczg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:12.827975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.827956 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs" pod "dae06927-a3bf-4a96-8da3-99871d1801a5" (UID: "dae06927-a3bf-4a96-8da3-99871d1801a5"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:12.828197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.828166 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs" pod "dae06927-a3bf-4a96-8da3-99871d1801a5" (UID: "dae06927-a3bf-4a96-8da3-99871d1801a5"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:12.926790 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.926766 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:12.926913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.926793 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:12.926913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.926807 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:12.926913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.926820 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdczg\" (UniqueName: \"kubernetes.io/projected/dae06927-a3bf-4a96-8da3-99871d1801a5-kube-api-access-hdczg\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:12.926913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.926834 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae06927-a3bf-4a96-8da3-99871d1801a5-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:12.926913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:12.926846 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/dae06927-a3bf-4a96-8da3-99871d1801a5-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:13.752007 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.751928 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.752445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.751939 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"dae06927-a3bf-4a96-8da3-99871d1801a5","Type":"ContainerDied","Data":"a9d8cedd7ab2796a422d8587a2bf9f94b88fc66c8b319216dafe0658350037b5"} Oct 30 18:57:13.752445 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.752061 6966 scope.go:122] "RemoveContainer" containerID="e238a819e4e4335537ac935d1ee9f5156d9c87ddd6599f72b70dc45b37253f7b" Oct 30 18:57:13.800302 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.800272 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:13.811685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.811653 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:13.842323 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.841985 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:13.842604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.842584 6966 cpu_manager.go:414] "containerMap: removing container" podUID="dae06927-a3bf-4a96-8da3-99871d1801a5" containerName="otc-container" Oct 30 18:57:13.842693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.842606 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae06927-a3bf-4a96-8da3-99871d1801a5" containerName="otc-container" Oct 30 18:57:13.842761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.842693 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae06927-a3bf-4a96-8da3-99871d1801a5" containerName="otc-container" Oct 30 18:57:13.853433 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.853409 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.868766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.868742 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:57:13.868933 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.868913 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:57:13.869070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.869039 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-p8bgw\"" Oct 30 18:57:13.869954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.869931 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:57:13.870557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.870536 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:13.870753 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.870617 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:57:13.878288 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.878268 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:57:13.937185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.937162 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p4tt\" (UniqueName: \"kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.937299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.937193 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.937341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.937323 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.937385 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.937369 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.937472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.937446 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:13.937524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:13.937488 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.037920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.037854 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p4tt\" (UniqueName: \"kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.037920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.037891 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.038111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.037961 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.038111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.037996 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.038111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.038032 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.038111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.038062 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.038894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.038869 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.040502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.040482 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.040614 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.040588 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.040854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.040836 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.041362 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.041345 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.060180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.060159 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p4tt\" (UniqueName: \"kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt\") pod \"data-science-collector-collector-1\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.166246 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.166226 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:14.463130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.463097 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:14.467126 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:57:14.467099 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafabb358_d0d2_4811_96ff_34733a16253e.slice/crio-4bc4db5f30fa6e2ca277eb90c22b8299b57b1e15554938558135a5d7a548ae4d WatchSource:0}: Error finding container 4bc4db5f30fa6e2ca277eb90c22b8299b57b1e15554938558135a5d7a548ae4d: Status 404 returned error can't find the container with id 4bc4db5f30fa6e2ca277eb90c22b8299b57b1e15554938558135a5d7a548ae4d Oct 30 18:57:14.469684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.469645 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:57:14.469771 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.469724 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:57:14.760848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.760774 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"afabb358-d0d2-4811-96ff-34733a16253e","Type":"ContainerStarted","Data":"9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56"} Oct 30 18:57:14.760848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.760827 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"afabb358-d0d2-4811-96ff-34733a16253e","Type":"ContainerStarted","Data":"4bc4db5f30fa6e2ca277eb90c22b8299b57b1e15554938558135a5d7a548ae4d"} Oct 30 18:57:14.801475 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:14.801424 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.8014115149999999 podStartE2EDuration="1.801411515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:14.80082669 +0000 UTC m=+3597.507012048" watchObservedRunningTime="2026-10-30 18:57:14.801411515 +0000 UTC m=+3597.507596872" Oct 30 18:57:15.411270 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:15.411237 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae06927-a3bf-4a96-8da3-99871d1801a5" path="/var/lib/kubelet/pods/dae06927-a3bf-4a96-8da3-99871d1801a5/volumes" Oct 30 18:57:17.141149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:17.141111 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:17.198882 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:17.198848 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:57:19.083760 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.083720 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h"] Oct 30 18:57:19.084417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.084212 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="kube-rbac-proxy" containerID="cri-o://fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" gracePeriod=30 Oct 30 18:57:19.084417 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.084263 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="prom-label-proxy" containerID="cri-o://c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" gracePeriod=30 Oct 30 18:57:19.084601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.084435 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="method-gate" containerID="cri-o://4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" gracePeriod=30 Oct 30 18:57:19.105784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.105761 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6"] Oct 30 18:57:19.105999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.105955 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" podUID="869b5b61-27b3-4883-a591-9398e2e2ad11" containerName="korrel8r" containerID="cri-o://abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714" gracePeriod=30 Oct 30 18:57:19.358002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.357972 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:57:19.382210 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.382183 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:57:19.384224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.383133 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls\" (UniqueName: \"kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls\") pod \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " Oct 30 18:57:19.384224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.383219 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config\" (UniqueName: \"kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config\") pod \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " Oct 30 18:57:19.384224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.383313 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw\" (UniqueName: \"kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw\") pod \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\" (UID: \"ad2f0836-8daf-41f7-a61f-48fd8012f46b\") " Oct 30 18:57:19.384224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.383924 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config" pod "ad2f0836-8daf-41f7-a61f-48fd8012f46b" (UID: "ad2f0836-8daf-41f7-a61f-48fd8012f46b"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.385941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.385917 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw" pod "ad2f0836-8daf-41f7-a61f-48fd8012f46b" (UID: "ad2f0836-8daf-41f7-a61f-48fd8012f46b"). InnerVolumeSpecName "kube-api-access-gkrtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.387744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.387717 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls" pod "ad2f0836-8daf-41f7-a61f-48fd8012f46b" (UID: "ad2f0836-8daf-41f7-a61f-48fd8012f46b"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.484550 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.484506 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s\" (UniqueName: \"kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s\") pod \"869b5b61-27b3-4883-a591-9398e2e2ad11\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " Oct 30 18:57:19.484728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.484623 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config\" (UniqueName: \"kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config\") pod \"869b5b61-27b3-4883-a591-9398e2e2ad11\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " Oct 30 18:57:19.484728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.484694 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls\" (UniqueName: \"kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls\") pod \"869b5b61-27b3-4883-a591-9398e2e2ad11\" (UID: \"869b5b61-27b3-4883-a591-9398e2e2ad11\") " Oct 30 18:57:19.485130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.485108 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/ad2f0836-8daf-41f7-a61f-48fd8012f46b-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.485260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.485134 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ad2f0836-8daf-41f7-a61f-48fd8012f46b-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.485260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.485149 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkrtw\" (UniqueName: \"kubernetes.io/projected/ad2f0836-8daf-41f7-a61f-48fd8012f46b-kube-api-access-gkrtw\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.486779 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.486548 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config" pod "869b5b61-27b3-4883-a591-9398e2e2ad11" (UID: "869b5b61-27b3-4883-a591-9398e2e2ad11"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.491947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.491905 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s" pod "869b5b61-27b3-4883-a591-9398e2e2ad11" (UID: "869b5b61-27b3-4883-a591-9398e2e2ad11"). InnerVolumeSpecName "kube-api-access-b8b6s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.491947 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.491981 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls" pod "869b5b61-27b3-4883-a591-9398e2e2ad11" (UID: "869b5b61-27b3-4883-a591-9398e2e2ad11"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.551907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.551831 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:19.556887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.556864 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:57:19.557188 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.557159 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="prometheus" containerID="cri-o://d8594b3cdaefeae1717a54ed3ebbe1949188aeced82042d2ba7ba1bd7223ec66" gracePeriod=600 Oct 30 18:57:19.557296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.557194 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="thanos-sidecar" containerID="cri-o://997fe1ddf00150ccb162ee4b2a6ffbb81ec68b6bea4955565ea0b5adbfee7f1f" gracePeriod=600 Oct 30 18:57:19.557296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.557286 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="config-reloader" containerID="cri-o://ca6f71fdd1d961cac9bb5aab29bf5edbf66c1de371246ec0f9e4574b3151e8ab" gracePeriod=600 Oct 30 18:57:19.567981 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.567956 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:57:19.568180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.568161 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="alertmanager" containerID="cri-o://30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a" gracePeriod=120 Oct 30 18:57:19.568304 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.568289 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="config-reloader" containerID="cri-o://922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b" gracePeriod=120 Oct 30 18:57:19.586202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.586136 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/869b5b61-27b3-4883-a591-9398e2e2ad11-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.586202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.586175 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/869b5b61-27b3-4883-a591-9398e2e2ad11-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.586202 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.586190 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8b6s\" (UniqueName: \"kubernetes.io/projected/869b5b61-27b3-4883-a591-9398e2e2ad11-kube-api-access-b8b6s\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.764545 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.764508 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:19.792031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.792003 6966 generic.go:356] "Generic (PLEG): container finished" podID="43ab7571-0308-484b-af6c-1e91865d28fa" containerID="922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b" exitCode=0 Oct 30 18:57:19.792183 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.792072 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerDied","Data":"922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b"} Oct 30 18:57:19.793505 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.793448 6966 generic.go:356] "Generic (PLEG): container finished" podID="869b5b61-27b3-4883-a591-9398e2e2ad11" containerID="abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714" exitCode=0 Oct 30 18:57:19.793602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.793502 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" event={"ID":"869b5b61-27b3-4883-a591-9398e2e2ad11","Type":"ContainerDied","Data":"abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714"} Oct 30 18:57:19.793602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.793514 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" Oct 30 18:57:19.793602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.793541 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6" event={"ID":"869b5b61-27b3-4883-a591-9398e2e2ad11","Type":"ContainerDied","Data":"27311dc116b811ebcf2c860d2b0f572448ab986faf3e370dfa3adbaeaea71923"} Oct 30 18:57:19.793602 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.793568 6966 scope.go:122] "RemoveContainer" containerID="abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714" Oct 30 18:57:19.796657 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.796634 6966 generic.go:356] "Generic (PLEG): container finished" podID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerID="997fe1ddf00150ccb162ee4b2a6ffbb81ec68b6bea4955565ea0b5adbfee7f1f" exitCode=0 Oct 30 18:57:19.796825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.796658 6966 generic.go:356] "Generic (PLEG): container finished" podID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerID="ca6f71fdd1d961cac9bb5aab29bf5edbf66c1de371246ec0f9e4574b3151e8ab" exitCode=0 Oct 30 18:57:19.796825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.796673 6966 generic.go:356] "Generic (PLEG): container finished" podID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerID="d8594b3cdaefeae1717a54ed3ebbe1949188aeced82042d2ba7ba1bd7223ec66" exitCode=0 Oct 30 18:57:19.796825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.796706 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerDied","Data":"997fe1ddf00150ccb162ee4b2a6ffbb81ec68b6bea4955565ea0b5adbfee7f1f"} Oct 30 18:57:19.796825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.796761 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerDied","Data":"ca6f71fdd1d961cac9bb5aab29bf5edbf66c1de371246ec0f9e4574b3151e8ab"} Oct 30 18:57:19.796825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.796793 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerDied","Data":"d8594b3cdaefeae1717a54ed3ebbe1949188aeced82042d2ba7ba1bd7223ec66"} Oct 30 18:57:19.799050 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799024 6966 generic.go:356] "Generic (PLEG): container finished" podID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerID="c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" exitCode=0 Oct 30 18:57:19.799050 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799045 6966 generic.go:356] "Generic (PLEG): container finished" podID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerID="4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" exitCode=0 Oct 30 18:57:19.799216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799056 6966 generic.go:356] "Generic (PLEG): container finished" podID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerID="fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" exitCode=0 Oct 30 18:57:19.799216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799085 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerDied","Data":"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2"} Oct 30 18:57:19.799216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799104 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" Oct 30 18:57:19.799216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799143 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerDied","Data":"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453"} Oct 30 18:57:19.799216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799182 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerDied","Data":"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f"} Oct 30 18:57:19.799427 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.799220 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h" event={"ID":"ad2f0836-8daf-41f7-a61f-48fd8012f46b","Type":"ContainerDied","Data":"71e6a80394b2605855c171d7e0cb3241bc91c5505435d1d250c05860983f0594"} Oct 30 18:57:19.804049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.804029 6966 scope.go:122] "RemoveContainer" containerID="abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714" Oct 30 18:57:19.804351 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:19.804325 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714\": container with ID starting with abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714 not found: ID does not exist" containerID="abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714" Oct 30 18:57:19.804423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.804362 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714"} err="failed to get container status \"abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714\": rpc error: code = NotFound desc = could not find container \"abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714\": container with ID starting with abf0207b979446ae81a5c3d2a9323b4ac2979ad268c86d23fc2b08293cad9714 not found: ID does not exist" Oct 30 18:57:19.804423 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.804387 6966 scope.go:122] "RemoveContainer" containerID="c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" Oct 30 18:57:19.812886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.812869 6966 scope.go:122] "RemoveContainer" containerID="4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" Oct 30 18:57:19.820695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.820408 6966 scope.go:122] "RemoveContainer" containerID="fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" Oct 30 18:57:19.828311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.828294 6966 scope.go:122] "RemoveContainer" containerID="c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" Oct 30 18:57:19.828641 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:19.828611 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": container with ID starting with c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2 not found: ID does not exist" containerID="c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" Oct 30 18:57:19.828747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.828650 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2"} err="failed to get container status \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": rpc error: code = NotFound desc = could not find container \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": container with ID starting with c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2 not found: ID does not exist" Oct 30 18:57:19.828747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.828672 6966 scope.go:122] "RemoveContainer" containerID="4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" Oct 30 18:57:19.828968 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:19.828951 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": container with ID starting with 4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453 not found: ID does not exist" containerID="4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" Oct 30 18:57:19.829012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.828974 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453"} err="failed to get container status \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": rpc error: code = NotFound desc = could not find container \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": container with ID starting with 4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453 not found: ID does not exist" Oct 30 18:57:19.829012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.828998 6966 scope.go:122] "RemoveContainer" containerID="fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" Oct 30 18:57:19.829248 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:19.829233 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": container with ID starting with fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f not found: ID does not exist" containerID="fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" Oct 30 18:57:19.829297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.829252 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f"} err="failed to get container status \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": rpc error: code = NotFound desc = could not find container \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": container with ID starting with fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f not found: ID does not exist" Oct 30 18:57:19.829297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.829264 6966 scope.go:122] "RemoveContainer" containerID="c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" Oct 30 18:57:19.829619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.829598 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2"} err="failed to get container status \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": rpc error: code = NotFound desc = could not find container \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": container with ID starting with c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2 not found: ID does not exist" Oct 30 18:57:19.829714 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.829619 6966 scope.go:122] "RemoveContainer" containerID="4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" Oct 30 18:57:19.829895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.829874 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453"} err="failed to get container status \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": rpc error: code = NotFound desc = could not find container \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": container with ID starting with 4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453 not found: ID does not exist" Oct 30 18:57:19.829945 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.829896 6966 scope.go:122] "RemoveContainer" containerID="fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" Oct 30 18:57:19.830136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830118 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f"} err="failed to get container status \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": rpc error: code = NotFound desc = could not find container \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": container with ID starting with fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f not found: ID does not exist" Oct 30 18:57:19.830182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830137 6966 scope.go:122] "RemoveContainer" containerID="c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2" Oct 30 18:57:19.830364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830347 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2"} err="failed to get container status \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": rpc error: code = NotFound desc = could not find container \"c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2\": container with ID starting with c1640333c67c6d1caa38b3c23c1d31dfd651949e7a49d0ba78d200427be742c2 not found: ID does not exist" Oct 30 18:57:19.830432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830364 6966 scope.go:122] "RemoveContainer" containerID="4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453" Oct 30 18:57:19.830604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830589 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453"} err="failed to get container status \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": rpc error: code = NotFound desc = could not find container \"4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453\": container with ID starting with 4d825dfeb68f0fe3e8d3c4db570ff949fb65964bf8da9fa58bcfc7a649acd453 not found: ID does not exist" Oct 30 18:57:19.830666 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830604 6966 scope.go:122] "RemoveContainer" containerID="fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f" Oct 30 18:57:19.830849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.830829 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f"} err="failed to get container status \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": rpc error: code = NotFound desc = could not find container \"fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f\": container with ID starting with fc4b685706e8c3b20624309fd9cef08e91ac03deaabcb335c3369e4f29a0ab4f not found: ID does not exist" Oct 30 18:57:19.843332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.843287 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6"] Oct 30 18:57:19.849866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.849842 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-799l6"] Oct 30 18:57:19.880584 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.880560 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h"] Oct 30 18:57:19.884324 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.884298 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-v6v4h"] Oct 30 18:57:19.906297 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.906279 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:19.970987 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.970960 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:19.990075 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990040 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990094 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out\" (UniqueName: \"kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990132 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990172 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990204 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990301 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990332 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990393 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990486 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990451 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990499 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990543 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990579 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990622 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config\") pod \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\" (UID: \"837885a4-532e-488d-b7a9-a79fcbbe4ae3\") " Oct 30 18:57:19.990717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990683 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.990996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.990955 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:19.991225 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.991198 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.991830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.991509 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:19.993300 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.993235 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.993409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.993320 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:19.993409 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.993347 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.994137 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.994118 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.994239 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.994197 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.994899 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.994873 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "kube-api-access-9v4tf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:19.995142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.995121 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.995244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.995137 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:19.996744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:19.996724 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:57:20.007716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.007691 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config" pod "837885a4-532e-488d-b7a9-a79fcbbe4ae3" (UID: "837885a4-532e-488d-b7a9-a79fcbbe4ae3"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:20.091687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091663 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.091687 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091685 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091694 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091703 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/837885a4-532e-488d-b7a9-a79fcbbe4ae3-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091712 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091721 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9v4tf\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-kube-api-access-9v4tf\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091744 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091755 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091765 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091774 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/837885a4-532e-488d-b7a9-a79fcbbe4ae3-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091784 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/837885a4-532e-488d-b7a9-a79fcbbe4ae3-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.092032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.091793 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/837885a4-532e-488d-b7a9-a79fcbbe4ae3-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.102968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.102951 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:57:20.193242 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.193209 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 18:57:20.207632 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.207595 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:20.207799 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.207778 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="afabb358-d0d2-4811-96ff-34733a16253e" containerName="otc-container" containerID="cri-o://9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56" gracePeriod=30 Oct 30 18:57:20.351474 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.351389 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:20.465374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.465352 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:20.495766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.495742 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs\") pod \"afabb358-d0d2-4811-96ff-34733a16253e\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " Oct 30 18:57:20.495906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.495776 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal\" (UniqueName: \"kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal\") pod \"afabb358-d0d2-4811-96ff-34733a16253e\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " Oct 30 18:57:20.495906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.495809 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs\") pod \"afabb358-d0d2-4811-96ff-34733a16253e\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " Oct 30 18:57:20.495906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.495833 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs\") pod \"afabb358-d0d2-4811-96ff-34733a16253e\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " Oct 30 18:57:20.496074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.495929 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert\") pod \"afabb358-d0d2-4811-96ff-34733a16253e\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " Oct 30 18:57:20.496074 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.495993 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt\" (UniqueName: \"kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt\") pod \"afabb358-d0d2-4811-96ff-34733a16253e\" (UID: \"afabb358-d0d2-4811-96ff-34733a16253e\") " Oct 30 18:57:20.496216 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.496187 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal" pod "afabb358-d0d2-4811-96ff-34733a16253e" (UID: "afabb358-d0d2-4811-96ff-34733a16253e"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:57:20.496337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.496309 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/afabb358-d0d2-4811-96ff-34733a16253e-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.498090 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.498063 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert" pod "afabb358-d0d2-4811-96ff-34733a16253e" (UID: "afabb358-d0d2-4811-96ff-34733a16253e"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:20.498398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.498373 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs" pod "afabb358-d0d2-4811-96ff-34733a16253e" (UID: "afabb358-d0d2-4811-96ff-34733a16253e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:20.498398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.498383 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt" pod "afabb358-d0d2-4811-96ff-34733a16253e" (UID: "afabb358-d0d2-4811-96ff-34733a16253e"). InnerVolumeSpecName "kube-api-access-2p4tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:20.498563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.498448 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs" pod "afabb358-d0d2-4811-96ff-34733a16253e" (UID: "afabb358-d0d2-4811-96ff-34733a16253e"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:20.498563 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.498454 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs" pod "afabb358-d0d2-4811-96ff-34733a16253e" (UID: "afabb358-d0d2-4811-96ff-34733a16253e"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:20.554380 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.554351 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:20.597552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.597525 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.597552 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.597546 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p4tt\" (UniqueName: \"kubernetes.io/projected/afabb358-d0d2-4811-96ff-34733a16253e-kube-api-access-2p4tt\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.597716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.597558 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.597716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.597569 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.597716 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.597577 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/afabb358-d0d2-4811-96ff-34733a16253e-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:20.806617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.806528 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"837885a4-532e-488d-b7a9-a79fcbbe4ae3","Type":"ContainerDied","Data":"81d51098bdbbc9fe965b4a35c045052371a3b0c20b39b803182f9a0ca63bd9d6"} Oct 30 18:57:20.806617 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.806614 6966 scope.go:122] "RemoveContainer" containerID="997fe1ddf00150ccb162ee4b2a6ffbb81ec68b6bea4955565ea0b5adbfee7f1f" Oct 30 18:57:20.806899 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.806543 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:20.808115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.808089 6966 generic.go:356] "Generic (PLEG): container finished" podID="afabb358-d0d2-4811-96ff-34733a16253e" containerID="9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56" exitCode=0 Oct 30 18:57:20.808240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.808132 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:20.808240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.808167 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"afabb358-d0d2-4811-96ff-34733a16253e","Type":"ContainerDied","Data":"9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56"} Oct 30 18:57:20.808240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.808236 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"afabb358-d0d2-4811-96ff-34733a16253e","Type":"ContainerDied","Data":"4bc4db5f30fa6e2ca277eb90c22b8299b57b1e15554938558135a5d7a548ae4d"} Oct 30 18:57:20.816250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.816232 6966 scope.go:122] "RemoveContainer" containerID="ca6f71fdd1d961cac9bb5aab29bf5edbf66c1de371246ec0f9e4574b3151e8ab" Oct 30 18:57:20.825118 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.825101 6966 scope.go:122] "RemoveContainer" containerID="d8594b3cdaefeae1717a54ed3ebbe1949188aeced82042d2ba7ba1bd7223ec66" Oct 30 18:57:20.833073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.833056 6966 scope.go:122] "RemoveContainer" containerID="b9fb37ab93fc27c86e8e38498b9acfc9bf8e84cc65f81735cca6cf4f2664bafd" Oct 30 18:57:20.840968 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.840953 6966 scope.go:122] "RemoveContainer" containerID="9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56" Oct 30 18:57:20.848439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.848422 6966 scope.go:122] "RemoveContainer" containerID="9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56" Oct 30 18:57:20.848719 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:20.848701 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56\": container with ID starting with 9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56 not found: ID does not exist" containerID="9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56" Oct 30 18:57:20.848773 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.848726 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56"} err="failed to get container status \"9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56\": rpc error: code = NotFound desc = could not find container \"9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56\": container with ID starting with 9715b29dc91f82e7e2a13f73b3a97005e701d771919845f618971fc6da5f2b56 not found: ID does not exist" Oct 30 18:57:20.855369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.855343 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:20.859930 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.859911 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:20.893826 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.893803 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:57:20.904705 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:20.904682 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:57:21.187387 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:21.187354 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:21.403226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:21.403140 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:21.411416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:21.411389 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" path="/var/lib/kubelet/pods/837885a4-532e-488d-b7a9-a79fcbbe4ae3/volumes" Oct 30 18:57:21.411918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:21.411904 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869b5b61-27b3-4883-a591-9398e2e2ad11" path="/var/lib/kubelet/pods/869b5b61-27b3-4883-a591-9398e2e2ad11/volumes" Oct 30 18:57:21.412549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:21.412533 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" path="/var/lib/kubelet/pods/ad2f0836-8daf-41f7-a61f-48fd8012f46b/volumes" Oct 30 18:57:21.412978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:21.412965 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afabb358-d0d2-4811-96ff-34733a16253e" path="/var/lib/kubelet/pods/afabb358-d0d2-4811-96ff-34733a16253e/volumes" Oct 30 18:57:22.552260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:22.552226 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.150:9093/-/ready\": dial tcp 10.134.0.150:9093: connect: connection refused" Oct 30 18:57:23.916492 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:23.916449 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:24.921055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:24.921026 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:27.551954 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:27.551916 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.150:9093/-/ready\": dial tcp 10.134.0.150:9093: connect: connection refused" Oct 30 18:57:29.814277 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.814255 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:29.859908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.859851 6966 generic.go:356] "Generic (PLEG): container finished" podID="43ab7571-0308-484b-af6c-1e91865d28fa" containerID="30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a" exitCode=0 Oct 30 18:57:29.860013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.859915 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:29.860013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.859930 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerDied","Data":"30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a"} Oct 30 18:57:29.860013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.859994 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"43ab7571-0308-484b-af6c-1e91865d28fa","Type":"ContainerDied","Data":"723462b3f4e911dd0c5774d6deb4f10863a8881114ba5b2582b445481133f5a9"} Oct 30 18:57:29.860110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.860022 6966 scope.go:122] "RemoveContainer" containerID="922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b" Oct 30 18:57:29.866227 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866204 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.866314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866256 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.866314 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866302 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.866416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866328 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.866416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866358 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.866416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866395 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.866607 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866449 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume\") pod \"43ab7571-0308-484b-af6c-1e91865d28fa\" (UID: \"43ab7571-0308-484b-af6c-1e91865d28fa\") " Oct 30 18:57:29.867011 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.866979 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:29.868581 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.868559 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "kube-api-access-fhdzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:29.869186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.868930 6966 scope.go:122] "RemoveContainer" containerID="30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a" Oct 30 18:57:29.869186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.869040 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:57:29.869186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.869110 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:57:29.869186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.869183 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:29.874724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.874621 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:29.883435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.883410 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config" pod "43ab7571-0308-484b-af6c-1e91865d28fa" (UID: "43ab7571-0308-484b-af6c-1e91865d28fa"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:57:29.889394 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.889372 6966 scope.go:122] "RemoveContainer" containerID="b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69" Oct 30 18:57:29.898170 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.898150 6966 scope.go:122] "RemoveContainer" containerID="922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b" Oct 30 18:57:29.898434 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:29.898413 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b\": container with ID starting with 922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b not found: ID does not exist" containerID="922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b" Oct 30 18:57:29.898537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.898445 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b"} err="failed to get container status \"922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b\": rpc error: code = NotFound desc = could not find container \"922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b\": container with ID starting with 922e8ace8aee3629ddc5ca9fe7735e0cb465c9483a0a3b40ae02d42554cbe18b not found: ID does not exist" Oct 30 18:57:29.898537 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.898481 6966 scope.go:122] "RemoveContainer" containerID="30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a" Oct 30 18:57:29.898759 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:29.898741 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a\": container with ID starting with 30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a not found: ID does not exist" containerID="30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a" Oct 30 18:57:29.898796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.898765 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a"} err="failed to get container status \"30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a\": rpc error: code = NotFound desc = could not find container \"30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a\": container with ID starting with 30b5744356441924f9a79548e03eb0250159dcbaf2472d296ea8f3721daa261a not found: ID does not exist" Oct 30 18:57:29.898796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.898779 6966 scope.go:122] "RemoveContainer" containerID="b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69" Oct 30 18:57:29.898988 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:29.898971 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69\": container with ID starting with b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69 not found: ID does not exist" containerID="b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69" Oct 30 18:57:29.899046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.898995 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69"} err="failed to get container status \"b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69\": rpc error: code = NotFound desc = could not find container \"b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69\": container with ID starting with b2c6568ef1056b3228c7afe76b98e09a1870893de085183af08580ba4e9a5f69 not found: ID does not exist" Oct 30 18:57:29.967996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.967971 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.967996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.967992 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.967996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.968001 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.968153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.968011 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/43ab7571-0308-484b-af6c-1e91865d28fa-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.968153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.968021 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/43ab7571-0308-484b-af6c-1e91865d28fa-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.968153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.968029 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:29.968153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:29.968038 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhdzr\" (UniqueName: \"kubernetes.io/projected/43ab7571-0308-484b-af6c-1e91865d28fa-kube-api-access-fhdzr\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:57:30.203777 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:30.203750 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:57:30.214997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:30.214970 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:57:31.410784 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:31.410753 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" path="/var/lib/kubelet/pods/43ab7571-0308-484b-af6c-1e91865d28fa/volumes" Oct 30 18:57:47.198853 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:57:47.198811 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:57:49.541444 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.541408 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:49.767214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.767179 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:49.823644 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.823614 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t"] Oct 30 18:57:49.823986 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.823968 6966 cpu_manager.go:414] "containerMap: removing container" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="config-reloader" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.823986 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="config-reloader" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824007 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="kube-rbac-proxy" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824013 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="kube-rbac-proxy" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824022 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="prom-label-proxy" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824028 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="prom-label-proxy" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824041 6966 cpu_manager.go:414] "containerMap: removing container" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="alertmanager" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824046 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="alertmanager" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824060 6966 cpu_manager.go:414] "containerMap: removing container" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="prometheus" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824068 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="prometheus" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824077 6966 cpu_manager.go:414] "containerMap: removing container" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="init-config-reloader" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824082 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="init-config-reloader" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824088 6966 cpu_manager.go:414] "containerMap: removing container" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="thanos-sidecar" Oct 30 18:57:49.824097 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824093 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="thanos-sidecar" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824124 6966 cpu_manager.go:414] "containerMap: removing container" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="init-config-reloader" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824129 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="init-config-reloader" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824142 6966 cpu_manager.go:414] "containerMap: removing container" podUID="869b5b61-27b3-4883-a591-9398e2e2ad11" containerName="korrel8r" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824149 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869b5b61-27b3-4883-a591-9398e2e2ad11" containerName="korrel8r" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824161 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="method-gate" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824165 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="method-gate" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824171 6966 cpu_manager.go:414] "containerMap: removing container" podUID="afabb358-d0d2-4811-96ff-34733a16253e" containerName="otc-container" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824175 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afabb358-d0d2-4811-96ff-34733a16253e" containerName="otc-container" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824196 6966 cpu_manager.go:414] "containerMap: removing container" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="config-reloader" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824201 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="config-reloader" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824279 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="config-reloader" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824293 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="config-reloader" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824304 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="kube-rbac-proxy" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824312 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="prom-label-proxy" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824321 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="43ab7571-0308-484b-af6c-1e91865d28fa" containerName="alertmanager" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824327 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="prometheus" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824331 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="837885a4-532e-488d-b7a9-a79fcbbe4ae3" containerName="thanos-sidecar" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824338 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="869b5b61-27b3-4883-a591-9398e2e2ad11" containerName="korrel8r" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824345 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad2f0836-8daf-41f7-a61f-48fd8012f46b" containerName="method-gate" Oct 30 18:57:49.824761 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.824350 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="afabb358-d0d2-4811-96ff-34733a16253e" containerName="otc-container" Oct 30 18:57:49.827866 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.827850 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:49.855747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.855727 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t"] Oct 30 18:57:49.861084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.861062 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:57:49.861084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.861076 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:57:49.861229 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.861155 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:57:49.861271 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.861256 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:57:49.865589 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.865574 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-klwm4\"" Oct 30 18:57:49.868102 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.868085 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg"] Oct 30 18:57:49.871045 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.871029 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:49.897124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.897102 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg"] Oct 30 18:57:49.916998 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.916972 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:49.917103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.917014 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm244\" (UniqueName: \"kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:49.917103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.917037 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:49.962153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.962125 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:57:49.962255 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.962190 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-4dpkk\"" Oct 30 18:57:49.962327 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.962312 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:57:49.986144 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:49.986114 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.018293 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.018269 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.018401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.018300 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.018401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.018319 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr296\" (UniqueName: \"kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.018401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.018378 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.018577 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.018559 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm244\" (UniqueName: \"kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.018619 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.018601 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.019254 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.019233 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.020776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.020758 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.041601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.041578 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm244\" (UniqueName: \"kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244\") pod \"data-science-thanos-querier-proxy-6998ff8b98-45k8t\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.119369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.119305 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.119369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.119346 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.119369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.119364 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr296\" (UniqueName: \"kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.120155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.120139 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.121499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.121481 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.141819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.141800 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr296\" (UniqueName: \"kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296\") pod \"korrel8r-5cbcc89b8f-vglhg\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.149624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.149605 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:57:50.183280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.183254 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.446501 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.446412 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t"] Oct 30 18:57:50.451123 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:57:50.451098 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118972ed_d68f_4b98_b634_6987ed9436bd.slice/crio-4f0afc1d027ef8521569b71bca0f13c181838962d33bd52b39fa762bb694128b WatchSource:0}: Error finding container 4f0afc1d027ef8521569b71bca0f13c181838962d33bd52b39fa762bb694128b: Status 404 returned error can't find the container with id 4f0afc1d027ef8521569b71bca0f13c181838962d33bd52b39fa762bb694128b Oct 30 18:57:50.485332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.485308 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg"] Oct 30 18:57:50.490955 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:57:50.490930 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod900046a6_f6e0_4c32_ad77_06fcab5cf714.slice/crio-72b600c9779ad03ea8ba3d7f1c9f65c1454b0ff7ad892fd9a541d9499e51f582 WatchSource:0}: Error finding container 72b600c9779ad03ea8ba3d7f1c9f65c1454b0ff7ad892fd9a541d9499e51f582: Status 404 returned error can't find the container with id 72b600c9779ad03ea8ba3d7f1c9f65c1454b0ff7ad892fd9a541d9499e51f582 Oct 30 18:57:50.498506 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.498452 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.701436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.701377 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.913502 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.913448 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:50.971752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.971653 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" event={"ID":"900046a6-f6e0-4c32-ad77-06fcab5cf714","Type":"ContainerStarted","Data":"34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d"} Oct 30 18:57:50.971752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.971748 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" event={"ID":"900046a6-f6e0-4c32-ad77-06fcab5cf714","Type":"ContainerStarted","Data":"72b600c9779ad03ea8ba3d7f1c9f65c1454b0ff7ad892fd9a541d9499e51f582"} Oct 30 18:57:50.972012 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.971781 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:57:50.973969 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.973941 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerStarted","Data":"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90"} Oct 30 18:57:50.974079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.974003 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerStarted","Data":"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef"} Oct 30 18:57:50.974079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.974030 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerStarted","Data":"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879"} Oct 30 18:57:50.974079 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:50.974053 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerStarted","Data":"4f0afc1d027ef8521569b71bca0f13c181838962d33bd52b39fa762bb694128b"} Oct 30 18:57:51.010872 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.010827 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" podStartSLOduration=2.0108144 podStartE2EDuration="2.0108144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:51.009637989 +0000 UTC m=+3633.715823346" watchObservedRunningTime="2026-10-30 18:57:51.0108144 +0000 UTC m=+3633.716999753" Oct 30 18:57:51.050943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.050904 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" podStartSLOduration=2.050892694 podStartE2EDuration="2.050892694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:51.049499404 +0000 UTC m=+3633.755684761" watchObservedRunningTime="2026-10-30 18:57:51.050892694 +0000 UTC m=+3633.757078051" Oct 30 18:57:51.353367 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.353337 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:51.357560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.357537 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.378740 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.378712 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:51.392477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.392439 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:57:51.431978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.431951 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.432147 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.432018 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.432226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.432136 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.432226 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.432178 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swbf5\" (UniqueName: \"kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.432346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.432253 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.432346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.432278 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.447800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.447767 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.449275 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.449253 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:57:51.449575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.449538 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lcs2t\"" Oct 30 18:57:51.449575 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.449562 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:57:51.449804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.449583 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:57:51.449804 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.449674 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:57:51.532713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.532685 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.532888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.532722 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.532888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.532743 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swbf5\" (UniqueName: \"kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.532888 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.532872 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.533139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.532910 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.533139 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.532966 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.533569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.533540 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.535793 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.535772 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.535906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.535844 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.535906 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.535848 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.536033 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.535974 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.554966 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.554938 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swbf5\" (UniqueName: \"kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5\") pod \"data-science-collector-collector-1\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.656299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.656238 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.673240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.673215 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:57:51.854957 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.854923 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:51.973943 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.973873 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:57:51.979045 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:57:51.979022 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59a4015f_9896_4b00_949d_2102f387c001.slice/crio-f8e287276676cc3ab9b6a19a4ccbc5b3017bf3d177007706c58392dfff43bad7 WatchSource:0}: Error finding container f8e287276676cc3ab9b6a19a4ccbc5b3017bf3d177007706c58392dfff43bad7: Status 404 returned error can't find the container with id f8e287276676cc3ab9b6a19a4ccbc5b3017bf3d177007706c58392dfff43bad7 Oct 30 18:57:51.981330 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.981303 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:57:51.981407 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:51.981357 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:57:52.986603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:52.986569 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a4015f-9896-4b00-949d-2102f387c001","Type":"ContainerStarted","Data":"5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12"} Oct 30 18:57:52.986993 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:52.986623 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a4015f-9896-4b00-949d-2102f387c001","Type":"ContainerStarted","Data":"f8e287276676cc3ab9b6a19a4ccbc5b3017bf3d177007706c58392dfff43bad7"} Oct 30 18:57:53.030261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:53.030211 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.030194351 podStartE2EDuration="2.030194351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:57:53.028284206 +0000 UTC m=+3635.734469563" watchObservedRunningTime="2026-10-30 18:57:53.030194351 +0000 UTC m=+3635.736379701" Oct 30 18:57:58.593721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:58.593687 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:58.833401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:58.833365 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.051404 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.051315 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.120902 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.120870 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:57:59.129522 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.129496 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.143758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.143717 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:57:59.143918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.143881 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:57:59.143997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.143937 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:57:59.144306 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.144285 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:57:59.154622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.154583 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-j9qcn\"" Oct 30 18:57:59.157015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.156994 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:57:59.186693 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.186664 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:57:59.190651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.190632 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.210913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.210889 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:57:59.210913 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.210889 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:57:59.211039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.211012 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:57:59.221016 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.220992 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:57:59.244130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244068 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:57:59.244259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244136 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:57:59.244259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244139 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:57:59.244259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244140 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-qnsch\"" Oct 30 18:57:59.244508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244487 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:57:59.244612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244552 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:57:59.244683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.244665 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:57:59.260682 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.260654 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.295897 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.295871 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprlk\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296011 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.295905 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296011 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.295926 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296018 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296087 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296067 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296094 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296117 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296174 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296212 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296260 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296292 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296278 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4474\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296296 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296358 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296341 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296428 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296363 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296448 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.296528 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296506 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296565 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296525 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296565 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296541 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.296634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.296581 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397321 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397358 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397400 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397419 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397435 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397451 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397527 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397501 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397532 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprlk\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397557 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397587 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397618 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397653 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397679 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397700 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397724 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397755 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397771 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397788 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4474\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.397805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397804 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.398388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.397849 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.398388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.398042 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.399871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.399843 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.400942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.400912 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.401122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.401096 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.401346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.401319 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.401479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.401417 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.401696 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.401662 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.402192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.402166 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.402789 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.402748 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.402995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.402957 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.403199 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.403172 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.403285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.403216 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.403440 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.403421 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.403547 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.403500 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.403794 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.403775 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.404594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.404577 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.421352 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.421328 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprlk\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.421430 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.421368 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4474\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:57:59.448795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.448775 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:57:59.460348 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.460321 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:57:59.778030 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:57:59.777948 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:57:59.782332 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:57:59.782299 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod690a62a8_b7e5_42f0_adc5_83ac7039ffee.slice/crio-cca7541de76b36fd07a1c76689e2ccd656e6cab3b03db807a742a5f21db9914e WatchSource:0}: Error finding container cca7541de76b36fd07a1c76689e2ccd656e6cab3b03db807a742a5f21db9914e: Status 404 returned error can't find the container with id cca7541de76b36fd07a1c76689e2ccd656e6cab3b03db807a742a5f21db9914e Oct 30 18:58:00.022286 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:00.022247 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerStarted","Data":"cca7541de76b36fd07a1c76689e2ccd656e6cab3b03db807a742a5f21db9914e"} Oct 30 18:58:01.029692 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:01.029600 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerStarted","Data":"a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd"} Oct 30 18:58:01.982184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:01.982155 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:58:03.834839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:03.834807 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:03.862039 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:03.862011 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:03.936099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:03.936076 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:03.936185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:03.936166 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:03.949815 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:03.949794 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:04.048280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:04.048247 6966 generic.go:356] "Generic (PLEG): container finished" podID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerID="a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd" exitCode=0 Oct 30 18:58:04.048406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:04.048307 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerDied","Data":"a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd"} Oct 30 18:58:04.052903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:04.052876 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:04.078898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:04.078869 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:04.311630 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:04.311601 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:04.654137 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:58:04.654110 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c3aeb9e_463c_4873_b8cb_7872906141a7.slice/crio-b7164c138d2a6ec61fb9d1868fce8dff3d7b520084eed1cc26cc154326374750 WatchSource:0}: Error finding container b7164c138d2a6ec61fb9d1868fce8dff3d7b520084eed1cc26cc154326374750: Status 404 returned error can't find the container with id b7164c138d2a6ec61fb9d1868fce8dff3d7b520084eed1cc26cc154326374750 Oct 30 18:58:04.661813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:04.661669 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:58:05.055379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:05.055353 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerStarted","Data":"5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270"} Oct 30 18:58:05.055728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:05.055416 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerStarted","Data":"c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812"} Oct 30 18:58:05.055728 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:05.055623 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:58:05.056940 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:05.056918 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerStarted","Data":"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d"} Oct 30 18:58:05.057055 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:05.056981 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerStarted","Data":"b7164c138d2a6ec61fb9d1868fce8dff3d7b520084eed1cc26cc154326374750"} Oct 30 18:58:05.108015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:05.107964 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=2.423342376 podStartE2EDuration="6.107947575s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.684605199s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:05.106786142 +0000 UTC m=+3647.812971499" watchObservedRunningTime="2026-10-30 18:58:05.107947575 +0000 UTC m=+3647.814132933" Oct 30 18:58:09.079870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:09.079835 6966 generic.go:356] "Generic (PLEG): container finished" podID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerID="79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d" exitCode=0 Oct 30 18:58:09.080236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:09.079907 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerDied","Data":"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d"} Oct 30 18:58:10.087368 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:10.087337 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerStarted","Data":"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27"} Oct 30 18:58:10.087781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:10.087403 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerStarted","Data":"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4"} Oct 30 18:58:10.087781 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:10.087429 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerStarted","Data":"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d"} Oct 30 18:58:10.160145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:10.160085 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=7.573609754 podStartE2EDuration="11.160068679s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586458925s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:57:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:10.151505535 +0000 UTC m=+3652.857690894" watchObservedRunningTime="2026-10-30 18:58:10.160068679 +0000 UTC m=+3652.866254038" Oct 30 18:58:11.064223 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:11.064190 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:58:14.312603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:14.312569 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:17.199178 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:17.199138 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:58:19.312706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:19.312673 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:19.317806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:19.317783 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:20.144961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:20.144934 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:21.004499 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:21.004423 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:21.242426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:21.242392 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:21.489564 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:21.489532 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:22.659124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:22.659082 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:22.659640 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:22.659303 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="59a4015f-9896-4b00-949d-2102f387c001" containerName="otc-container" containerID="cri-o://5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12" gracePeriod=30 Oct 30 18:58:22.906805 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:22.906758 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:23.135870 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:23.135834 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:24.048999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:24.048961 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:25.006511 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.006486 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.034128 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.034094 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:25.122743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.122675 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5\" (UniqueName: \"kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5\") pod \"59a4015f-9896-4b00-949d-2102f387c001\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " Oct 30 18:58:25.123186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.122772 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert\") pod \"59a4015f-9896-4b00-949d-2102f387c001\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " Oct 30 18:58:25.123186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.122844 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal\") pod \"59a4015f-9896-4b00-949d-2102f387c001\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " Oct 30 18:58:25.123186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.122913 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs\") pod \"59a4015f-9896-4b00-949d-2102f387c001\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " Oct 30 18:58:25.123186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.122951 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs\") pod \"59a4015f-9896-4b00-949d-2102f387c001\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " Oct 30 18:58:25.123186 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.122987 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs\") pod \"59a4015f-9896-4b00-949d-2102f387c001\" (UID: \"59a4015f-9896-4b00-949d-2102f387c001\") " Oct 30 18:58:25.123491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.123238 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal" pod "59a4015f-9896-4b00-949d-2102f387c001" (UID: "59a4015f-9896-4b00-949d-2102f387c001"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:25.125059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.125025 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5" pod "59a4015f-9896-4b00-949d-2102f387c001" (UID: "59a4015f-9896-4b00-949d-2102f387c001"). InnerVolumeSpecName "kube-api-access-swbf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:25.125201 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.125076 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert" pod "59a4015f-9896-4b00-949d-2102f387c001" (UID: "59a4015f-9896-4b00-949d-2102f387c001"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:25.125344 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.125325 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs" pod "59a4015f-9896-4b00-949d-2102f387c001" (UID: "59a4015f-9896-4b00-949d-2102f387c001"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:25.125413 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.125388 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs" pod "59a4015f-9896-4b00-949d-2102f387c001" (UID: "59a4015f-9896-4b00-949d-2102f387c001"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:25.125525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.125422 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs" pod "59a4015f-9896-4b00-949d-2102f387c001" (UID: "59a4015f-9896-4b00-949d-2102f387c001"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:25.166160 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.166134 6966 generic.go:356] "Generic (PLEG): container finished" podID="59a4015f-9896-4b00-949d-2102f387c001" containerID="5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12" exitCode=0 Oct 30 18:58:25.166262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.166181 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.166262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.166191 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a4015f-9896-4b00-949d-2102f387c001","Type":"ContainerDied","Data":"5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12"} Oct 30 18:58:25.166262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.166233 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"59a4015f-9896-4b00-949d-2102f387c001","Type":"ContainerDied","Data":"f8e287276676cc3ab9b6a19a4ccbc5b3017bf3d177007706c58392dfff43bad7"} Oct 30 18:58:25.166262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.166253 6966 scope.go:122] "RemoveContainer" containerID="5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12" Oct 30 18:58:25.175046 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.175030 6966 scope.go:122] "RemoveContainer" containerID="5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12" Oct 30 18:58:25.175301 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:25.175281 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12\": container with ID starting with 5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12 not found: ID does not exist" containerID="5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12" Oct 30 18:58:25.175342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.175308 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12"} err="failed to get container status \"5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12\": rpc error: code = NotFound desc = could not find container \"5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12\": container with ID starting with 5da79f3a5b0907ddb40339be94bb7eb7f1a26f65549f44ac85491a0bfa3c4c12 not found: ID does not exist" Oct 30 18:58:25.223937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.223908 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.223937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.223935 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.224072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.223950 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.224072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.223964 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swbf5\" (UniqueName: \"kubernetes.io/projected/59a4015f-9896-4b00-949d-2102f387c001-kube-api-access-swbf5\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.224072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.223978 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/59a4015f-9896-4b00-949d-2102f387c001-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.224072 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.223992 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/59a4015f-9896-4b00-949d-2102f387c001-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:25.225814 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.225794 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:25.231842 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.231820 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:25.266707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.266683 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:25.267150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.267127 6966 cpu_manager.go:414] "containerMap: removing container" podUID="59a4015f-9896-4b00-949d-2102f387c001" containerName="otc-container" Oct 30 18:58:25.267150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.267147 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59a4015f-9896-4b00-949d-2102f387c001" containerName="otc-container" Oct 30 18:58:25.267317 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.267287 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="59a4015f-9896-4b00-949d-2102f387c001" containerName="otc-container" Oct 30 18:58:25.271684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.271666 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.285994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.285971 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lcs2t\"" Oct 30 18:58:25.286108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.285995 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:58:25.286108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.286035 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:58:25.286108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.286039 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:58:25.286108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.286071 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:58:25.286283 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.286162 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:58:25.293810 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.293789 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:25.412021 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.411957 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59a4015f-9896-4b00-949d-2102f387c001" path="/var/lib/kubelet/pods/59a4015f-9896-4b00-949d-2102f387c001/volumes" Oct 30 18:58:25.425230 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.425203 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.425375 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.425270 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwzk2\" (UniqueName: \"kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.425451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.425376 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.425451 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.425415 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.425604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.425530 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.425604 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.425559 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.526342 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.526317 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwzk2\" (UniqueName: \"kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.526489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.526349 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.526489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.526369 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.526489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.526389 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.526489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.526405 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.526489 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.526481 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.527219 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.527190 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.529006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.528985 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.529124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.529055 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.529257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.529238 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.529296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.529247 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.548825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.548802 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwzk2\" (UniqueName: \"kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2\") pod \"data-science-collector-collector-1\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.584746 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.584722 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:25.882791 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.882759 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:25.887825 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:58:25.887799 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84ffa4b8_1447_4191_a8be_b615f0e83086.slice/crio-abf4665de65489e117867899a7cfc97b6016755e12b753564dcd5174ccccd546 WatchSource:0}: Error finding container abf4665de65489e117867899a7cfc97b6016755e12b753564dcd5174ccccd546: Status 404 returned error can't find the container with id abf4665de65489e117867899a7cfc97b6016755e12b753564dcd5174ccccd546 Oct 30 18:58:25.890113 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.890079 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:58:25.890189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:25.890140 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:58:26.175044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:26.174963 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"84ffa4b8-1447-4191-a8be-b615f0e83086","Type":"ContainerStarted","Data":"6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c"} Oct 30 18:58:26.175044 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:26.175020 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"84ffa4b8-1447-4191-a8be-b615f0e83086","Type":"ContainerStarted","Data":"abf4665de65489e117867899a7cfc97b6016755e12b753564dcd5174ccccd546"} Oct 30 18:58:26.216813 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:26.216757 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.216744339 podStartE2EDuration="1.216744339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:26.216710205 +0000 UTC m=+3668.922895575" watchObservedRunningTime="2026-10-30 18:58:26.216744339 +0000 UTC m=+3668.922929689" Oct 30 18:58:36.020494 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:36.020437 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:37.715710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:37.715684 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:38.046515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:38.046407 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:38.265695 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:38.265659 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.437000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.436969 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:39.437524 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.437184 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="84ffa4b8-1447-4191-a8be-b615f0e83086" containerName="otc-container" containerID="cri-o://6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c" gracePeriod=30 Oct 30 18:58:39.684002 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.683966 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:39.689709 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.688765 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:39.844364 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844332 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal\" (UniqueName: \"kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal\") pod \"84ffa4b8-1447-4191-a8be-b615f0e83086\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " Oct 30 18:58:39.844570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844401 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs\") pod \"84ffa4b8-1447-4191-a8be-b615f0e83086\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " Oct 30 18:58:39.844570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844437 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert\") pod \"84ffa4b8-1447-4191-a8be-b615f0e83086\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " Oct 30 18:58:39.844570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844502 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs\") pod \"84ffa4b8-1447-4191-a8be-b615f0e83086\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " Oct 30 18:58:39.844570 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844557 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs\") pod \"84ffa4b8-1447-4191-a8be-b615f0e83086\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " Oct 30 18:58:39.844806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844684 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2\" (UniqueName: \"kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2\") pod \"84ffa4b8-1447-4191-a8be-b615f0e83086\" (UID: \"84ffa4b8-1447-4191-a8be-b615f0e83086\") " Oct 30 18:58:39.844806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.844793 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal" pod "84ffa4b8-1447-4191-a8be-b615f0e83086" (UID: "84ffa4b8-1447-4191-a8be-b615f0e83086"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:39.845022 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.845003 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/84ffa4b8-1447-4191-a8be-b615f0e83086-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:39.847219 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.847191 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs" pod "84ffa4b8-1447-4191-a8be-b615f0e83086" (UID: "84ffa4b8-1447-4191-a8be-b615f0e83086"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:39.847339 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.847224 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs" pod "84ffa4b8-1447-4191-a8be-b615f0e83086" (UID: "84ffa4b8-1447-4191-a8be-b615f0e83086"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:39.847391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.847342 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2" pod "84ffa4b8-1447-4191-a8be-b615f0e83086" (UID: "84ffa4b8-1447-4191-a8be-b615f0e83086"). InnerVolumeSpecName "kube-api-access-bwzk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:39.847391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.847357 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert" pod "84ffa4b8-1447-4191-a8be-b615f0e83086" (UID: "84ffa4b8-1447-4191-a8be-b615f0e83086"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:39.847553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.847532 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs" pod "84ffa4b8-1447-4191-a8be-b615f0e83086" (UID: "84ffa4b8-1447-4191-a8be-b615f0e83086"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:39.946130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.946069 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:39.946130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.946093 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwzk2\" (UniqueName: \"kubernetes.io/projected/84ffa4b8-1447-4191-a8be-b615f0e83086-kube-api-access-bwzk2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:39.946130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.946104 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:39.946130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.946113 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:39.946130 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.946123 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84ffa4b8-1447-4191-a8be-b615f0e83086-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:39.961101 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:39.961072 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:40.247605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.247511 6966 generic.go:356] "Generic (PLEG): container finished" podID="84ffa4b8-1447-4191-a8be-b615f0e83086" containerID="6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c" exitCode=0 Oct 30 18:58:40.247605 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.247549 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"84ffa4b8-1447-4191-a8be-b615f0e83086","Type":"ContainerDied","Data":"6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c"} Oct 30 18:58:40.247840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.247561 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.247840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.247645 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"84ffa4b8-1447-4191-a8be-b615f0e83086","Type":"ContainerDied","Data":"abf4665de65489e117867899a7cfc97b6016755e12b753564dcd5174ccccd546"} Oct 30 18:58:40.247840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.247678 6966 scope.go:122] "RemoveContainer" containerID="6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c" Oct 30 18:58:40.259425 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.259408 6966 scope.go:122] "RemoveContainer" containerID="6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c" Oct 30 18:58:40.259739 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:40.259707 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c\": container with ID starting with 6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c not found: ID does not exist" containerID="6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c" Oct 30 18:58:40.259862 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.259743 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c"} err="failed to get container status \"6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c\": rpc error: code = NotFound desc = could not find container \"6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c\": container with ID starting with 6972aaa14f65f7ab4540e4055866d8b551f639f1e6fb4828c5b2086af813898c not found: ID does not exist" Oct 30 18:58:40.296659 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.296635 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:40.307679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.307650 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:40.339052 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.339024 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:40.339597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.339578 6966 cpu_manager.go:414] "containerMap: removing container" podUID="84ffa4b8-1447-4191-a8be-b615f0e83086" containerName="otc-container" Oct 30 18:58:40.339597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.339597 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84ffa4b8-1447-4191-a8be-b615f0e83086" containerName="otc-container" Oct 30 18:58:40.339778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.339747 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="84ffa4b8-1447-4191-a8be-b615f0e83086" containerName="otc-container" Oct 30 18:58:40.349997 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.349971 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.364285 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.364263 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:58:40.364402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.364334 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:58:40.364481 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.364412 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-lcs2t\"" Oct 30 18:58:40.364525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.364512 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:58:40.364879 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.364862 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:58:40.364973 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.364959 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:58:40.366115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.366050 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:40.449015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.448988 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.449421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.449023 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.449421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.449093 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.449421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.449123 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.449421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.449150 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr7wd\" (UniqueName: \"kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.449421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.449197 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.550521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.550444 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.550521 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.550491 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.550715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.550526 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.550715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.550557 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.550715 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.550583 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr7wd\" (UniqueName: \"kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.550868 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.550787 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.551669 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.551642 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.553196 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.553171 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.553294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.553217 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.553294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.553219 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.553294 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.553219 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.572875 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.572850 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr7wd\" (UniqueName: \"kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd\") pod \"data-science-collector-collector-1\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.662655 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.662620 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:40.965720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.965677 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:40.969826 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:58:40.969793 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeeb4c759_0b18_46fc_9638_230ec1c7156b.slice/crio-e6b231597f0195d436bcf89ce4bd8dcc5b6dcd25644e28113055d45e32231b50 WatchSource:0}: Error finding container e6b231597f0195d436bcf89ce4bd8dcc5b6dcd25644e28113055d45e32231b50: Status 404 returned error can't find the container with id e6b231597f0195d436bcf89ce4bd8dcc5b6dcd25644e28113055d45e32231b50 Oct 30 18:58:40.972164 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.972134 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:58:40.972250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:40.972191 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:58:41.255975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:41.255881 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eeb4c759-0b18-46fc-9638-230ec1c7156b","Type":"ContainerStarted","Data":"367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4"} Oct 30 18:58:41.255975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:41.255946 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eeb4c759-0b18-46fc-9638-230ec1c7156b","Type":"ContainerStarted","Data":"e6b231597f0195d436bcf89ce4bd8dcc5b6dcd25644e28113055d45e32231b50"} Oct 30 18:58:41.297508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:41.297376 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.297358966 podStartE2EDuration="1.297358966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:58:41.297192495 +0000 UTC m=+3684.003377849" watchObservedRunningTime="2026-10-30 18:58:41.297358966 +0000 UTC m=+3684.003544327" Oct 30 18:58:41.412515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:41.412455 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84ffa4b8-1447-4191-a8be-b615f0e83086" path="/var/lib/kubelet/pods/84ffa4b8-1447-4191-a8be-b615f0e83086/volumes" Oct 30 18:58:46.041059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:46.041029 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:47.199602 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:47.199566 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:58:48.233679 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.233634 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t"] Oct 30 18:58:48.234257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.233858 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="kube-rbac-proxy" containerID="cri-o://56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" gracePeriod=30 Oct 30 18:58:48.234257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.233968 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="method-gate" containerID="cri-o://d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" gracePeriod=30 Oct 30 18:58:48.234257 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.234130 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="prom-label-proxy" containerID="cri-o://6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" gracePeriod=30 Oct 30 18:58:48.257421 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.257400 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg"] Oct 30 18:58:48.257638 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.257615 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" podUID="900046a6-f6e0-4c32-ad77-06fcab5cf714" containerName="korrel8r" containerID="cri-o://34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d" gracePeriod=30 Oct 30 18:58:48.505593 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.505530 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:58:48.539253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.539226 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:58:48.615343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.615313 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244\" (UniqueName: \"kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244\") pod \"118972ed-d68f-4b98-b634-6987ed9436bd\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " Oct 30 18:58:48.615717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.615692 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls\" (UniqueName: \"kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls\") pod \"118972ed-d68f-4b98-b634-6987ed9436bd\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " Oct 30 18:58:48.615831 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.615742 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config\" (UniqueName: \"kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config\") pod \"118972ed-d68f-4b98-b634-6987ed9436bd\" (UID: \"118972ed-d68f-4b98-b634-6987ed9436bd\") " Oct 30 18:58:48.616311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.616280 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config" pod "118972ed-d68f-4b98-b634-6987ed9436bd" (UID: "118972ed-d68f-4b98-b634-6987ed9436bd"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:48.618725 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.618686 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls" pod "118972ed-d68f-4b98-b634-6987ed9436bd" (UID: "118972ed-d68f-4b98-b634-6987ed9436bd"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:48.618963 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.618930 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244" pod "118972ed-d68f-4b98-b634-6987ed9436bd" (UID: "118972ed-d68f-4b98-b634-6987ed9436bd"). InnerVolumeSpecName "kube-api-access-jm244". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:48.700909 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.700873 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:58:48.701184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.701155 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="alertmanager" containerID="cri-o://c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812" gracePeriod=120 Oct 30 18:58:48.701282 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.701254 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="config-reloader" containerID="cri-o://5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270" gracePeriod=120 Oct 30 18:58:48.703415 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.703388 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:58:48.703752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.703695 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="prometheus" containerID="cri-o://ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" gracePeriod=600 Oct 30 18:58:48.703752 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.703717 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="thanos-sidecar" containerID="cri-o://8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" gracePeriod=600 Oct 30 18:58:48.703881 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.703704 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="config-reloader" containerID="cri-o://aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" gracePeriod=600 Oct 30 18:58:48.705243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.705211 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:48.716284 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716264 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296\" (UniqueName: \"kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296\") pod \"900046a6-f6e0-4c32-ad77-06fcab5cf714\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " Oct 30 18:58:48.716391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716306 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config\" (UniqueName: \"kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config\") pod \"900046a6-f6e0-4c32-ad77-06fcab5cf714\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " Oct 30 18:58:48.716391 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716364 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls\" (UniqueName: \"kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls\") pod \"900046a6-f6e0-4c32-ad77-06fcab5cf714\" (UID: \"900046a6-f6e0-4c32-ad77-06fcab5cf714\") " Oct 30 18:58:48.716708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716643 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm244\" (UniqueName: \"kubernetes.io/projected/118972ed-d68f-4b98-b634-6987ed9436bd-kube-api-access-jm244\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.716708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716662 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/118972ed-d68f-4b98-b634-6987ed9436bd-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.716708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716692 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/118972ed-d68f-4b98-b634-6987ed9436bd-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.716856 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.716777 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config" pod "900046a6-f6e0-4c32-ad77-06fcab5cf714" (UID: "900046a6-f6e0-4c32-ad77-06fcab5cf714"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:48.718390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.718365 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296" pod "900046a6-f6e0-4c32-ad77-06fcab5cf714" (UID: "900046a6-f6e0-4c32-ad77-06fcab5cf714"). InnerVolumeSpecName "kube-api-access-wr296". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:48.718498 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.718392 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls" pod "900046a6-f6e0-4c32-ad77-06fcab5cf714" (UID: "900046a6-f6e0-4c32-ad77-06fcab5cf714"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:48.818004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.817978 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/900046a6-f6e0-4c32-ad77-06fcab5cf714-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.818004 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.818003 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr296\" (UniqueName: \"kubernetes.io/projected/900046a6-f6e0-4c32-ad77-06fcab5cf714-kube-api-access-wr296\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.818218 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.818014 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/900046a6-f6e0-4c32-ad77-06fcab5cf714-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:48.932435 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.932407 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:48.995962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:48.995941 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:49.119775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119700 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119744 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119783 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119821 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119852 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119888 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119928 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119957 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.119996 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.119985 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.120401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.120030 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.120401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.120072 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.120401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.120132 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.120401 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.120168 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474\") pod \"7c3aeb9e-463c-4873-b8cb-7872906141a7\" (UID: \"7c3aeb9e-463c-4873-b8cb-7872906141a7\") " Oct 30 18:58:49.120658 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.120622 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.121236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.121105 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.121236 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.121186 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.122637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.122587 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.122637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.122608 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.124392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.124345 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.124392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.124345 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "kube-api-access-p4474". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.124392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.124363 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.124392 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.124351 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.124747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.124429 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.124747 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.124623 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.125848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.125829 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:58:49.136234 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.136209 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config" pod "7c3aeb9e-463c-4873-b8cb-7872906141a7" (UID: "7c3aeb9e-463c-4873-b8cb-7872906141a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.221854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221832 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4474\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-kube-api-access-p4474\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221869 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221879 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221889 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221898 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221907 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221917 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221925 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7c3aeb9e-463c-4873-b8cb-7872906141a7-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221934 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.221941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221943 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.222248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221953 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7c3aeb9e-463c-4873-b8cb-7872906141a7-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.222248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221964 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7c3aeb9e-463c-4873-b8cb-7872906141a7-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.222248 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.221972 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7c3aeb9e-463c-4873-b8cb-7872906141a7-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.232887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.232870 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:58:49.303063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.303026 6966 generic.go:356] "Generic (PLEG): container finished" podID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerID="5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270" exitCode=0 Oct 30 18:58:49.303514 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.303102 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerDied","Data":"5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270"} Oct 30 18:58:49.306185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306161 6966 generic.go:356] "Generic (PLEG): container finished" podID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerID="8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" exitCode=0 Oct 30 18:58:49.306185 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306183 6966 generic.go:356] "Generic (PLEG): container finished" podID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerID="aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" exitCode=0 Oct 30 18:58:49.306378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306194 6966 generic.go:356] "Generic (PLEG): container finished" podID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerID="ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" exitCode=0 Oct 30 18:58:49.306378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306284 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerDied","Data":"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27"} Oct 30 18:58:49.306378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306293 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:58:49.306378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306345 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerDied","Data":"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4"} Oct 30 18:58:49.306553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306386 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerDied","Data":"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d"} Oct 30 18:58:49.306553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306425 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7c3aeb9e-463c-4873-b8cb-7872906141a7","Type":"ContainerDied","Data":"b7164c138d2a6ec61fb9d1868fce8dff3d7b520084eed1cc26cc154326374750"} Oct 30 18:58:49.306553 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.306486 6966 scope.go:122] "RemoveContainer" containerID="8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" Oct 30 18:58:49.309900 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.309810 6966 generic.go:356] "Generic (PLEG): container finished" podID="900046a6-f6e0-4c32-ad77-06fcab5cf714" containerID="34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d" exitCode=0 Oct 30 18:58:49.311154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.309942 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" event={"ID":"900046a6-f6e0-4c32-ad77-06fcab5cf714","Type":"ContainerDied","Data":"34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d"} Oct 30 18:58:49.311154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.310113 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" Oct 30 18:58:49.311154 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.310439 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg" event={"ID":"900046a6-f6e0-4c32-ad77-06fcab5cf714","Type":"ContainerDied","Data":"72b600c9779ad03ea8ba3d7f1c9f65c1454b0ff7ad892fd9a541d9499e51f582"} Oct 30 18:58:49.315437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315412 6966 generic.go:356] "Generic (PLEG): container finished" podID="118972ed-d68f-4b98-b634-6987ed9436bd" containerID="6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" exitCode=0 Oct 30 18:58:49.315437 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315436 6966 generic.go:356] "Generic (PLEG): container finished" podID="118972ed-d68f-4b98-b634-6987ed9436bd" containerID="d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" exitCode=0 Oct 30 18:58:49.315641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315449 6966 generic.go:356] "Generic (PLEG): container finished" podID="118972ed-d68f-4b98-b634-6987ed9436bd" containerID="56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" exitCode=0 Oct 30 18:58:49.315641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315484 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerDied","Data":"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90"} Oct 30 18:58:49.315641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315539 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerDied","Data":"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef"} Oct 30 18:58:49.315641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315576 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerDied","Data":"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879"} Oct 30 18:58:49.315641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315613 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" event={"ID":"118972ed-d68f-4b98-b634-6987ed9436bd","Type":"ContainerDied","Data":"4f0afc1d027ef8521569b71bca0f13c181838962d33bd52b39fa762bb694128b"} Oct 30 18:58:49.315910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.315707 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t" Oct 30 18:58:49.322726 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.322691 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 18:58:49.324374 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.324126 6966 scope.go:122] "RemoveContainer" containerID="aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" Oct 30 18:58:49.333176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.333153 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:49.333603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.333568 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="eeb4c759-0b18-46fc-9638-230ec1c7156b" containerName="otc-container" containerID="cri-o://367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4" gracePeriod=30 Oct 30 18:58:49.339015 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.338991 6966 scope.go:122] "RemoveContainer" containerID="ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" Oct 30 18:58:49.351253 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.351237 6966 scope.go:122] "RemoveContainer" containerID="79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d" Oct 30 18:58:49.366541 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.366518 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg"] Oct 30 18:58:49.372583 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.372511 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-vglhg"] Oct 30 18:58:49.412887 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.412859 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="900046a6-f6e0-4c32-ad77-06fcab5cf714" path="/var/lib/kubelet/pods/900046a6-f6e0-4c32-ad77-06fcab5cf714/volumes" Oct 30 18:58:49.423603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.423573 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:58:49.431708 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.431682 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:58:49.466837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.466815 6966 scope.go:122] "RemoveContainer" containerID="8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" Oct 30 18:58:49.467207 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.467173 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": container with ID starting with 8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27 not found: ID does not exist" containerID="8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" Oct 30 18:58:49.467301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.467218 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27"} err="failed to get container status \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": rpc error: code = NotFound desc = could not find container \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": container with ID starting with 8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27 not found: ID does not exist" Oct 30 18:58:49.467301 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.467244 6966 scope.go:122] "RemoveContainer" containerID="aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" Oct 30 18:58:49.467690 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.467668 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": container with ID starting with aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4 not found: ID does not exist" containerID="aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" Oct 30 18:58:49.467738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.467695 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4"} err="failed to get container status \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": rpc error: code = NotFound desc = could not find container \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": container with ID starting with aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4 not found: ID does not exist" Oct 30 18:58:49.467738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.467712 6966 scope.go:122] "RemoveContainer" containerID="ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" Oct 30 18:58:49.468059 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.468041 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": container with ID starting with ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d not found: ID does not exist" containerID="ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" Oct 30 18:58:49.468100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468063 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d"} err="failed to get container status \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": rpc error: code = NotFound desc = could not find container \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": container with ID starting with ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d not found: ID does not exist" Oct 30 18:58:49.468100 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468079 6966 scope.go:122] "RemoveContainer" containerID="79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d" Oct 30 18:58:49.468321 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.468305 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": container with ID starting with 79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d not found: ID does not exist" containerID="79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d" Oct 30 18:58:49.468369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468325 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d"} err="failed to get container status \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": rpc error: code = NotFound desc = could not find container \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": container with ID starting with 79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d not found: ID does not exist" Oct 30 18:58:49.468369 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468338 6966 scope.go:122] "RemoveContainer" containerID="8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" Oct 30 18:58:49.468670 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468634 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27"} err="failed to get container status \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": rpc error: code = NotFound desc = could not find container \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": container with ID starting with 8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27 not found: ID does not exist" Oct 30 18:58:49.468735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468676 6966 scope.go:122] "RemoveContainer" containerID="aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" Oct 30 18:58:49.468972 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468953 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4"} err="failed to get container status \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": rpc error: code = NotFound desc = could not find container \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": container with ID starting with aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4 not found: ID does not exist" Oct 30 18:58:49.469025 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.468973 6966 scope.go:122] "RemoveContainer" containerID="ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" Oct 30 18:58:49.469192 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469174 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d"} err="failed to get container status \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": rpc error: code = NotFound desc = could not find container \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": container with ID starting with ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d not found: ID does not exist" Oct 30 18:58:49.469244 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469192 6966 scope.go:122] "RemoveContainer" containerID="79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d" Oct 30 18:58:49.469388 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469370 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d"} err="failed to get container status \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": rpc error: code = NotFound desc = could not find container \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": container with ID starting with 79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d not found: ID does not exist" Oct 30 18:58:49.469433 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469387 6966 scope.go:122] "RemoveContainer" containerID="8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27" Oct 30 18:58:49.469627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469609 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27"} err="failed to get container status \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": rpc error: code = NotFound desc = could not find container \"8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27\": container with ID starting with 8153665e20f939d822c87addc88b3cf4eb5fa25c790c2179fe0e79b702873f27 not found: ID does not exist" Oct 30 18:58:49.469627 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469626 6966 scope.go:122] "RemoveContainer" containerID="aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4" Oct 30 18:58:49.469816 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469800 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4"} err="failed to get container status \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": rpc error: code = NotFound desc = could not find container \"aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4\": container with ID starting with aaadbc974dd2aadc72f46b2b03655e3d2e8999017eeb7a190ccd0e361d1fddd4 not found: ID does not exist" Oct 30 18:58:49.469865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469815 6966 scope.go:122] "RemoveContainer" containerID="ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d" Oct 30 18:58:49.469995 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469978 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d"} err="failed to get container status \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": rpc error: code = NotFound desc = could not find container \"ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d\": container with ID starting with ad55814b0fa358923206dc4e28951772d3fb7be8acdfcef9ba16dd46d527a08d not found: ID does not exist" Oct 30 18:58:49.470043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.469995 6966 scope.go:122] "RemoveContainer" containerID="79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d" Oct 30 18:58:49.470325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.470217 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d"} err="failed to get container status \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": rpc error: code = NotFound desc = could not find container \"79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d\": container with ID starting with 79cbc2404b2d7791c8d0838384e33e1287887c852db4696bc22c3f59e2a5fe5d not found: ID does not exist" Oct 30 18:58:49.470325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.470244 6966 scope.go:122] "RemoveContainer" containerID="34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d" Oct 30 18:58:49.475832 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.475787 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:49.492837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.491792 6966 scope.go:122] "RemoveContainer" containerID="34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d" Oct 30 18:58:49.493876 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.493835 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d\": container with ID starting with 34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d not found: ID does not exist" containerID="34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d" Oct 30 18:58:49.493961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.493871 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d"} err="failed to get container status \"34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d\": rpc error: code = NotFound desc = could not find container \"34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d\": container with ID starting with 34109cf049a41d9197476f0f96e738036f7759f9ab2f39db90d919fc6c56866d not found: ID does not exist" Oct 30 18:58:49.493961 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.493893 6966 scope.go:122] "RemoveContainer" containerID="6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" Oct 30 18:58:49.509312 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.509048 6966 scope.go:122] "RemoveContainer" containerID="d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" Oct 30 18:58:49.516737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.515037 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t"] Oct 30 18:58:49.517057 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.517012 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-45k8t"] Oct 30 18:58:49.522377 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.522296 6966 scope.go:122] "RemoveContainer" containerID="56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" Oct 30 18:58:49.530152 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.530134 6966 scope.go:122] "RemoveContainer" containerID="6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" Oct 30 18:58:49.530446 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.530428 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": container with ID starting with 6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90 not found: ID does not exist" containerID="6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" Oct 30 18:58:49.530533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.530453 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90"} err="failed to get container status \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": rpc error: code = NotFound desc = could not find container \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": container with ID starting with 6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90 not found: ID does not exist" Oct 30 18:58:49.530533 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.530488 6966 scope.go:122] "RemoveContainer" containerID="d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" Oct 30 18:58:49.530873 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.530851 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": container with ID starting with d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef not found: ID does not exist" containerID="d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" Oct 30 18:58:49.530999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.530880 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef"} err="failed to get container status \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": rpc error: code = NotFound desc = could not find container \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": container with ID starting with d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef not found: ID does not exist" Oct 30 18:58:49.530999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.530902 6966 scope.go:122] "RemoveContainer" containerID="56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" Oct 30 18:58:49.531191 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:49.531130 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": container with ID starting with 56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879 not found: ID does not exist" containerID="56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" Oct 30 18:58:49.531191 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.531159 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879"} err="failed to get container status \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": rpc error: code = NotFound desc = could not find container \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": container with ID starting with 56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879 not found: ID does not exist" Oct 30 18:58:49.531191 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.531178 6966 scope.go:122] "RemoveContainer" containerID="6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" Oct 30 18:58:49.531651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.531624 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90"} err="failed to get container status \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": rpc error: code = NotFound desc = could not find container \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": container with ID starting with 6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90 not found: ID does not exist" Oct 30 18:58:49.531769 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.531652 6966 scope.go:122] "RemoveContainer" containerID="d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" Oct 30 18:58:49.531920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.531900 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef"} err="failed to get container status \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": rpc error: code = NotFound desc = could not find container \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": container with ID starting with d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef not found: ID does not exist" Oct 30 18:58:49.532024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.531920 6966 scope.go:122] "RemoveContainer" containerID="56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" Oct 30 18:58:49.532148 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.532127 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879"} err="failed to get container status \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": rpc error: code = NotFound desc = could not find container \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": container with ID starting with 56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879 not found: ID does not exist" Oct 30 18:58:49.532240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.532149 6966 scope.go:122] "RemoveContainer" containerID="6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90" Oct 30 18:58:49.532395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.532377 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90"} err="failed to get container status \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": rpc error: code = NotFound desc = could not find container \"6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90\": container with ID starting with 6de544d6f10a462d31dbc69d73224538d0be7ad510fff8008e7aae79f06daa90 not found: ID does not exist" Oct 30 18:58:49.532596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.532396 6966 scope.go:122] "RemoveContainer" containerID="d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef" Oct 30 18:58:49.532755 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.532733 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef"} err="failed to get container status \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": rpc error: code = NotFound desc = could not find container \"d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef\": container with ID starting with d89403d21f1d53ef55840f4b1094255006dfe47d7cac88ae7dc30a19519c28ef not found: ID does not exist" Oct 30 18:58:49.532859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.532757 6966 scope.go:122] "RemoveContainer" containerID="56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879" Oct 30 18:58:49.533059 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.533029 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879"} err="failed to get container status \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": rpc error: code = NotFound desc = could not find container \"56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879\": container with ID starting with 56b0dcc353866009fe7bb70e74245f638f64c36907029c2dc74595600a575879 not found: ID does not exist" Oct 30 18:58:49.618062 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.618037 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:49.620224 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.620202 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:58:49.720334 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.720185 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:49.725953 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.725929 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.725972 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs\") pod \"eeb4c759-0b18-46fc-9638-230ec1c7156b\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " Oct 30 18:58:49.726108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726027 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert\") pod \"eeb4c759-0b18-46fc-9638-230ec1c7156b\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " Oct 30 18:58:49.726108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726072 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726101 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal\") pod \"eeb4c759-0b18-46fc-9638-230ec1c7156b\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " Oct 30 18:58:49.726333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726130 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726177 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs\") pod \"eeb4c759-0b18-46fc-9638-230ec1c7156b\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " Oct 30 18:58:49.726333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726222 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726275 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726333 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726312 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726349 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs\") pod \"eeb4c759-0b18-46fc-9638-230ec1c7156b\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " Oct 30 18:58:49.726626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726387 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config\") pod \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\" (UID: \"690a62a8-b7e5-42f0-adc5-83ac7039ffee\") " Oct 30 18:58:49.726626 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726417 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd\" (UniqueName: \"kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd\") pod \"eeb4c759-0b18-46fc-9638-230ec1c7156b\" (UID: \"eeb4c759-0b18-46fc-9638-230ec1c7156b\") " Oct 30 18:58:49.726786 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.726710 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal" pod "eeb4c759-0b18-46fc-9638-230ec1c7156b" (UID: "eeb4c759-0b18-46fc-9638-230ec1c7156b"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:58:49.727260 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.727200 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.729702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.729664 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd" pod "eeb4c759-0b18-46fc-9638-230ec1c7156b" (UID: "eeb4c759-0b18-46fc-9638-230ec1c7156b"). InnerVolumeSpecName "kube-api-access-rr7wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.729702 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.729691 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs" pod "eeb4c759-0b18-46fc-9638-230ec1c7156b" (UID: "eeb4c759-0b18-46fc-9638-230ec1c7156b"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.730819 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.730800 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.731280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.731086 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs" pod "eeb4c759-0b18-46fc-9638-230ec1c7156b" (UID: "eeb4c759-0b18-46fc-9638-230ec1c7156b"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.731280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.731229 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.731280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.731255 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:58:49.731280 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.731261 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "kube-api-access-kprlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:58:49.731707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.731669 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs" pod "eeb4c759-0b18-46fc-9638-230ec1c7156b" (UID: "eeb4c759-0b18-46fc-9638-230ec1c7156b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.733006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.732982 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert" pod "eeb4c759-0b18-46fc-9638-230ec1c7156b" (UID: "eeb4c759-0b18-46fc-9638-230ec1c7156b"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.735443 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.735418 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.742136 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.742118 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config" pod "690a62a8-b7e5-42f0-adc5-83ac7039ffee" (UID: "690a62a8-b7e5-42f0-adc5-83ac7039ffee"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:58:49.827378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827354 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827385 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827402 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/eeb4c759-0b18-46fc-9638-230ec1c7156b-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827416 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprlk\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-kube-api-access-kprlk\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827430 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827455 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827444 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827492 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/690a62a8-b7e5-42f0-adc5-83ac7039ffee-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827508 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827521 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827532 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/690a62a8-b7e5-42f0-adc5-83ac7039ffee-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827548 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr7wd\" (UniqueName: \"kubernetes.io/projected/eeb4c759-0b18-46fc-9638-230ec1c7156b-kube-api-access-rr7wd\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827561 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/690a62a8-b7e5-42f0-adc5-83ac7039ffee-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:49.827662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:49.827574 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/eeb4c759-0b18-46fc-9638-230ec1c7156b-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:58:50.326386 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.326348 6966 generic.go:356] "Generic (PLEG): container finished" podID="eeb4c759-0b18-46fc-9638-230ec1c7156b" containerID="367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4" exitCode=0 Oct 30 18:58:50.326855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.326493 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eeb4c759-0b18-46fc-9638-230ec1c7156b","Type":"ContainerDied","Data":"367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4"} Oct 30 18:58:50.326855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.326545 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"eeb4c759-0b18-46fc-9638-230ec1c7156b","Type":"ContainerDied","Data":"e6b231597f0195d436bcf89ce4bd8dcc5b6dcd25644e28113055d45e32231b50"} Oct 30 18:58:50.326855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.326573 6966 scope.go:122] "RemoveContainer" containerID="367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4" Oct 30 18:58:50.326855 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.326673 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:58:50.330396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.330366 6966 generic.go:356] "Generic (PLEG): container finished" podID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerID="c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812" exitCode=0 Oct 30 18:58:50.330613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.330492 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerDied","Data":"c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812"} Oct 30 18:58:50.330613 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.330553 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"690a62a8-b7e5-42f0-adc5-83ac7039ffee","Type":"ContainerDied","Data":"cca7541de76b36fd07a1c76689e2ccd656e6cab3b03db807a742a5f21db9914e"} Oct 30 18:58:50.331006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.330653 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 18:58:50.339049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.339027 6966 scope.go:122] "RemoveContainer" containerID="367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4" Oct 30 18:58:50.339334 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:50.339310 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4\": container with ID starting with 367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4 not found: ID does not exist" containerID="367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4" Oct 30 18:58:50.339426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.339343 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4"} err="failed to get container status \"367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4\": rpc error: code = NotFound desc = could not find container \"367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4\": container with ID starting with 367c9053294a9a94fad2a19710af5835c2fb086dcb7f66783425dc82e9e0d5f4 not found: ID does not exist" Oct 30 18:58:50.339426 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.339365 6966 scope.go:122] "RemoveContainer" containerID="5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270" Oct 30 18:58:50.347907 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.347891 6966 scope.go:122] "RemoveContainer" containerID="c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812" Oct 30 18:58:50.352895 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.352869 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:50.358549 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.358532 6966 scope.go:122] "RemoveContainer" containerID="a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd" Oct 30 18:58:50.368250 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.368234 6966 scope.go:122] "RemoveContainer" containerID="5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270" Oct 30 18:58:50.368518 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:50.368495 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270\": container with ID starting with 5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270 not found: ID does not exist" containerID="5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270" Oct 30 18:58:50.368598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.368524 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270"} err="failed to get container status \"5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270\": rpc error: code = NotFound desc = could not find container \"5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270\": container with ID starting with 5e5a401829c85eba36d323ca91d492401dcee41bb9667118a0627f46520e7270 not found: ID does not exist" Oct 30 18:58:50.368598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.368542 6966 scope.go:122] "RemoveContainer" containerID="c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812" Oct 30 18:58:50.368838 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:50.368817 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812\": container with ID starting with c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812 not found: ID does not exist" containerID="c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812" Oct 30 18:58:50.368936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.368846 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812"} err="failed to get container status \"c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812\": rpc error: code = NotFound desc = could not find container \"c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812\": container with ID starting with c2e81cbe81dd13d8640fe0fae713a4baf822256d34b6785f11df6119ecb18812 not found: ID does not exist" Oct 30 18:58:50.368936 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.368866 6966 scope.go:122] "RemoveContainer" containerID="a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd" Oct 30 18:58:50.369260 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:50.369216 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd\": container with ID starting with a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd not found: ID does not exist" containerID="a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd" Oct 30 18:58:50.369336 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.369255 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd"} err="failed to get container status \"a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd\": rpc error: code = NotFound desc = could not find container \"a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd\": container with ID starting with a9fe1b6c217b33c501a742d8c14c8530610d5d75f020c4ee979b434dd1c8e6dd not found: ID does not exist" Oct 30 18:58:50.392230 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.392205 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:50.400263 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.396172 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:58:50.426598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.426570 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:58:50.432937 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.432919 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 18:58:50.676169 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:50.676117 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:51.410920 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:51.410892 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" path="/var/lib/kubelet/pods/118972ed-d68f-4b98-b634-6987ed9436bd/volumes" Oct 30 18:58:51.411351 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:51.411337 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" path="/var/lib/kubelet/pods/690a62a8-b7e5-42f0-adc5-83ac7039ffee/volumes" Oct 30 18:58:51.412116 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:51.412101 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" path="/var/lib/kubelet/pods/7c3aeb9e-463c-4873-b8cb-7872906141a7/volumes" Oct 30 18:58:51.412592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:51.412580 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eeb4c759-0b18-46fc-9638-230ec1c7156b" path="/var/lib/kubelet/pods/eeb4c759-0b18-46fc-9638-230ec1c7156b/volumes" Oct 30 18:58:59.438298 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.438039 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.663911 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.663871 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.700603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700538 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn"] Oct 30 18:58:59.700871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700854 6966 cpu_manager.go:414] "containerMap: removing container" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="init-config-reloader" Oct 30 18:58:59.700871 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700868 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="init-config-reloader" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700884 6966 cpu_manager.go:414] "containerMap: removing container" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="kube-rbac-proxy" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700890 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="kube-rbac-proxy" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700896 6966 cpu_manager.go:414] "containerMap: removing container" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="alertmanager" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700901 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="alertmanager" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700915 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="prometheus" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700920 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="prometheus" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700926 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="thanos-sidecar" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700931 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="thanos-sidecar" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700940 6966 cpu_manager.go:414] "containerMap: removing container" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="prom-label-proxy" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700946 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="prom-label-proxy" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700958 6966 cpu_manager.go:414] "containerMap: removing container" podUID="900046a6-f6e0-4c32-ad77-06fcab5cf714" containerName="korrel8r" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700963 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="900046a6-f6e0-4c32-ad77-06fcab5cf714" containerName="korrel8r" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700968 6966 cpu_manager.go:414] "containerMap: removing container" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="method-gate" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700973 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="method-gate" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700979 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="config-reloader" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700983 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="config-reloader" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700990 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="init-config-reloader" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.700994 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="init-config-reloader" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701001 6966 cpu_manager.go:414] "containerMap: removing container" podUID="eeb4c759-0b18-46fc-9638-230ec1c7156b" containerName="otc-container" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701005 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eeb4c759-0b18-46fc-9638-230ec1c7156b" containerName="otc-container" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701012 6966 cpu_manager.go:414] "containerMap: removing container" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="config-reloader" Oct 30 18:58:59.701032 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701026 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="config-reloader" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701103 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="900046a6-f6e0-4c32-ad77-06fcab5cf714" containerName="korrel8r" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701110 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="method-gate" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701115 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="config-reloader" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701122 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="eeb4c759-0b18-46fc-9638-230ec1c7156b" containerName="otc-container" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701127 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="config-reloader" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701136 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="kube-rbac-proxy" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701144 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="690a62a8-b7e5-42f0-adc5-83ac7039ffee" containerName="alertmanager" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701153 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="prometheus" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701166 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c3aeb9e-463c-4873-b8cb-7872906141a7" containerName="thanos-sidecar" Oct 30 18:58:59.701942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.701173 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="118972ed-d68f-4b98-b634-6987ed9436bd" containerName="prom-label-proxy" Oct 30 18:58:59.704556 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.704538 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.725800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.725780 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn"] Oct 30 18:58:59.739848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.739826 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx"] Oct 30 18:58:59.743448 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.743432 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.745635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.745615 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 18:58:59.750008 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.749902 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:58:59.750008 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.749985 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-jbrs7\"" Oct 30 18:58:59.750149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.750003 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 18:58:59.750149 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.750026 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:58:59.763978 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.763956 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx"] Oct 30 18:58:59.798436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.798411 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drls2\" (UniqueName: \"kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.798543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.798443 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.798543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.798473 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.798633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.798551 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jtj2\" (UniqueName: \"kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.798633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.798617 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.798694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.798675 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.845852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.845823 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 18:58:59.845852 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.845823 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 18:58:59.845985 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.845906 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-x4cls\"" Oct 30 18:58:59.892208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.892183 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:58:59.899150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899133 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.899208 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899173 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drls2\" (UniqueName: \"kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.899243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899203 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.899243 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899227 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.899343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899263 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jtj2\" (UniqueName: \"kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.899343 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899302 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.899443 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:59.899411 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 30 18:58:59.899528 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:58:59.899498 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls podName:0decf7ad-ae2a-4cf2-b429-06e105f0100e nodeName:}" failed. No retries permitted until 2026-10-30 18:59:00.399454757 +0000 UTC m=+3703.105640092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls") pod "data-science-thanos-querier-proxy-6998ff8b98-mr9nn" (UID: "0decf7ad-ae2a-4cf2-b429-06e105f0100e") : secret "data-science-thanos-querier-proxy-tls" not found Oct 30 18:58:59.899885 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.899867 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.900049 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.900033 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.901839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.901820 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:58:59.922013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.921991 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drls2\" (UniqueName: \"kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:58:59.922115 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:58:59.922092 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jtj2\" (UniqueName: \"kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2\") pod \"korrel8r-5cbcc89b8f-ljsvx\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:59:00.055775 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.055700 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:59:00.119070 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.119037 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:00.335806 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.335775 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:00.351031 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.351010 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx"] Oct 30 18:59:00.356196 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:00.356168 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1742a849_4ba4_46c9_8df3_be18aac04c20.slice/crio-4385d904e8f529e2ca4834a7c11f37696974bc9cf98cc73644625370b51efdc5 WatchSource:0}: Error finding container 4385d904e8f529e2ca4834a7c11f37696974bc9cf98cc73644625370b51efdc5: Status 404 returned error can't find the container with id 4385d904e8f529e2ca4834a7c11f37696974bc9cf98cc73644625370b51efdc5 Oct 30 18:59:00.394178 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.394151 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" event={"ID":"1742a849-4ba4-46c9-8df3-be18aac04c20","Type":"ContainerStarted","Data":"4385d904e8f529e2ca4834a7c11f37696974bc9cf98cc73644625370b51efdc5"} Oct 30 18:59:00.403910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.403884 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:59:00.406191 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.406168 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-mr9nn\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:59:00.576637 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.576601 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:00.623063 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.623000 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:59:00.795472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.795436 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:00.931158 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:00.931081 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn"] Oct 30 18:59:00.935690 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:00.935659 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0decf7ad_ae2a_4cf2_b429_06e105f0100e.slice/crio-07391dd728f3618fc74f471974d5c23c4a9cc003e785cfc846c6983168fe7ed1 WatchSource:0}: Error finding container 07391dd728f3618fc74f471974d5c23c4a9cc003e785cfc846c6983168fe7ed1: Status 404 returned error can't find the container with id 07391dd728f3618fc74f471974d5c23c4a9cc003e785cfc846c6983168fe7ed1 Oct 30 18:59:01.165707 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.165677 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:01.169924 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.169899 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.195206 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.195149 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:01.200908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.200885 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:59:01.211390 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.211353 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xg9n\" (UniqueName: \"kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.211557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.211409 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.211557 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.211454 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.211689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.211575 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.211689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.211628 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.211689 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.211656 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.267869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.267829 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:59:01.267869 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.267829 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:59:01.268112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.267896 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:59:01.268112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.267903 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:59:01.268112 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.267978 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-bhcr4\"" Oct 30 18:59:01.277140 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.277113 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.312379 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.312352 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.312567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.312435 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.312567 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:01.312510 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-collector-collector-headless-tls: secret "data-science-collector-collector-headless-tls" not found Oct 30 18:59:01.312694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.312512 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.312694 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:01.312580 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs podName:31edd20e-95e3-4c28-9b56-56302ee95305 nodeName:}" failed. No retries permitted until 2026-10-30 18:59:01.812557686 +0000 UTC m=+3704.518743041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "otlp-tls-certs" (UniqueName: "kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs") pod "data-science-collector-collector-1" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305") : secret "data-science-collector-collector-headless-tls" not found Oct 30 18:59:01.312694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.312620 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.312694 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.312675 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xg9n\" (UniqueName: \"kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.312922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.312717 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.313378 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.313359 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.315023 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.315007 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.315081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.315013 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.315081 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.315044 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.337595 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.337572 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xg9n\" (UniqueName: \"kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.401685 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.401641 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" event={"ID":"1742a849-4ba4-46c9-8df3-be18aac04c20","Type":"ContainerStarted","Data":"52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665"} Oct 30 18:59:01.401828 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.401764 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:59:01.404084 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.404059 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerStarted","Data":"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc"} Oct 30 18:59:01.404193 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.404122 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerStarted","Data":"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97"} Oct 30 18:59:01.404193 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.404158 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerStarted","Data":"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76"} Oct 30 18:59:01.404299 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.404191 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerStarted","Data":"07391dd728f3618fc74f471974d5c23c4a9cc003e785cfc846c6983168fe7ed1"} Oct 30 18:59:01.440295 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.440257 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" podStartSLOduration=2.440246544 podStartE2EDuration="2.440246544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:01.440147966 +0000 UTC m=+3704.146333323" watchObservedRunningTime="2026-10-30 18:59:01.440246544 +0000 UTC m=+3704.146431900" Oct 30 18:59:01.483279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.483203 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" podStartSLOduration=2.483193159 podStartE2EDuration="2.483193159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:58:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:01.482385326 +0000 UTC m=+3704.188570685" watchObservedRunningTime="2026-10-30 18:59:01.483193159 +0000 UTC m=+3704.189378562" Oct 30 18:59:01.511311 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.511284 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.750651 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.750564 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:01.816923 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.816896 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:01.819353 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:01.819331 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:02.086078 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:02.086042 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:02.384092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:02.384001 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:02.389510 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:02.389476 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31edd20e_95e3_4c28_9b56_56302ee95305.slice/crio-f7f9212dc4d53e0b510b789e0830949b1d1b1a7c1d160795b974764f7dba318d WatchSource:0}: Error finding container f7f9212dc4d53e0b510b789e0830949b1d1b1a7c1d160795b974764f7dba318d: Status 404 returned error can't find the container with id f7f9212dc4d53e0b510b789e0830949b1d1b1a7c1d160795b974764f7dba318d Oct 30 18:59:02.391942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:02.391908 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:59:02.392013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:02.391970 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:59:02.411890 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:02.411862 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"31edd20e-95e3-4c28-9b56-56302ee95305","Type":"ContainerStarted","Data":"f7f9212dc4d53e0b510b789e0830949b1d1b1a7c1d160795b974764f7dba318d"} Oct 30 18:59:03.417737 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:03.417702 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"31edd20e-95e3-4c28-9b56-56302ee95305","Type":"ContainerStarted","Data":"00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06"} Oct 30 18:59:03.459076 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:03.459028 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=2.459014845 podStartE2EDuration="2.459014845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:03.458208797 +0000 UTC m=+3706.164394154" watchObservedRunningTime="2026-10-30 18:59:03.459014845 +0000 UTC m=+3706.165200202" Oct 30 18:59:08.243066 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.243031 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:08.479848 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.479809 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:08.698738 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.698701 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:08.776758 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.776729 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:59:08.786990 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.786947 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.806213 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.806187 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:59:08.816237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816207 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-wdmbl\"" Oct 30 18:59:08.816371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816273 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 18:59:08.816371 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816359 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 18:59:08.816508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816442 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:59:08.816582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816520 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 18:59:08.816582 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816560 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 18:59:08.816686 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816647 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 18:59:08.816742 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816696 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 18:59:08.816742 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816725 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 18:59:08.816839 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.816778 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 18:59:08.820662 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.820638 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:59:08.824319 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.824301 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.846928 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.846900 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:59:08.880272 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880246 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880283 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880309 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880389 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880347 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880525 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880499 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880567 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880534 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880603 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880579 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvxd8\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880600 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880641 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880623 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880722 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880707 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880732 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880747 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880774 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880824 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880893 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880846 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880894 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880926 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880913 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.880991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880947 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.880991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.880985 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpd4\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.908598 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.908577 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 18:59:08.908721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.908659 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 18:59:08.908891 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.908878 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 18:59:08.908934 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.908915 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 18:59:08.908984 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.908971 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-sb2qr\"" Oct 30 18:59:08.925350 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.925323 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:08.982106 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982049 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982106 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982088 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982122 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982147 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982179 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982213 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982254 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvxd8\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982278 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982316 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982312 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982348 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982375 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982398 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982428 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982452 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982512 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982552 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982581 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982627 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.982718 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.982657 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpd4\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.983488 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.983444 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.983609 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.983497 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.983672 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.983649 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.984013 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.983926 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.985155 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.985126 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.985841 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.985812 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.986335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.986286 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.986592 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.986567 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.986977 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.986951 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.987818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.987562 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.987818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.987769 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.987818 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.987773 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.988043 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.987935 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.988096 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.988049 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.988281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.988263 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:08.988318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.988298 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:08.989108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:08.989088 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:09.005189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:09.005165 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvxd8\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8\") pod \"alertmanager-data-science-monitoringstack-1\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:09.005258 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:09.005219 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpd4\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:09.142406 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:09.142381 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:09.459220 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:09.459182 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:59:09.463810 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:09.463781 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea4c466c_2b85_4cdd_9933_a45429afd9b5.slice/crio-9c5e8fb0d8ddf77ba31281ffb823419dd56625594fafc3088e6540807651a198 WatchSource:0}: Error finding container 9c5e8fb0d8ddf77ba31281ffb823419dd56625594fafc3088e6540807651a198: Status 404 returned error can't find the container with id 9c5e8fb0d8ddf77ba31281ffb823419dd56625594fafc3088e6540807651a198 Oct 30 18:59:10.457476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:10.457432 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerStarted","Data":"19a7635b43f9b8dc3893383fb6d58d2e55b34874c52ed50cf095441a330afb97"} Oct 30 18:59:10.457673 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:10.457508 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerStarted","Data":"9c5e8fb0d8ddf77ba31281ffb823419dd56625594fafc3088e6540807651a198"} Oct 30 18:59:12.414240 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:12.414211 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:59:13.476006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:13.475928 6966 generic.go:356] "Generic (PLEG): container finished" podID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerID="19a7635b43f9b8dc3893383fb6d58d2e55b34874c52ed50cf095441a330afb97" exitCode=0 Oct 30 18:59:13.476006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:13.475997 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerDied","Data":"19a7635b43f9b8dc3893383fb6d58d2e55b34874c52ed50cf095441a330afb97"} Oct 30 18:59:14.483429 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:14.483396 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerStarted","Data":"37def417bb0953c7e02bfbb856157cca59110989f6fba04ecd62b53b164f6be8"} Oct 30 18:59:14.483975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:14.483452 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerStarted","Data":"69879deb2f51f1e395cb6c2b7e4f50e02ed1637e7e19b8a9073725648b2bd26b"} Oct 30 18:59:14.483975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:14.483654 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:14.551477 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:14.551419 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podStartSLOduration=3.26616711 podStartE2EDuration="6.551402845s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.285235735s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:14.539743289 +0000 UTC m=+3717.245928646" watchObservedRunningTime="2026-10-30 18:59:14.551402845 +0000 UTC m=+3717.257588201" Oct 30 18:59:15.029410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.029376 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.056785 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.056751 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.130384 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.130352 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.130531 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.130498 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.143849 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.143824 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.244894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.244865 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.270491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.270446 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.407635 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.407599 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:15.755853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:15.755782 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:59:15.758978 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:15.758938 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7260339e_45a6_495a_980f_1e643e959bdc.slice/crio-1d55810be268f1142ca175ff3646203b8e46868e0da302829589fb23654f3893 WatchSource:0}: Error finding container 1d55810be268f1142ca175ff3646203b8e46868e0da302829589fb23654f3893: Status 404 returned error can't find the container with id 1d55810be268f1142ca175ff3646203b8e46868e0da302829589fb23654f3893 Oct 30 18:59:16.497000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:16.496913 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerStarted","Data":"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928"} Oct 30 18:59:16.497000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:16.496998 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerStarted","Data":"1d55810be268f1142ca175ff3646203b8e46868e0da302829589fb23654f3893"} Oct 30 18:59:17.201216 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:17.201181 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:59:20.490346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:20.490315 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 18:59:20.526267 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:20.526239 6966 generic.go:356] "Generic (PLEG): container finished" podID="7260339e-45a6-495a-980f-1e643e959bdc" containerID="6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928" exitCode=0 Oct 30 18:59:20.526410 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:20.526320 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerDied","Data":"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928"} Oct 30 18:59:21.534419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:21.534331 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerStarted","Data":"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9"} Oct 30 18:59:21.534419 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:21.534410 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerStarted","Data":"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847"} Oct 30 18:59:21.534922 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:21.534456 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerStarted","Data":"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b"} Oct 30 18:59:21.585517 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:21.585442 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=10.094758778 podStartE2EDuration="13.585432216s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.490673438s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:21.584955681 +0000 UTC m=+3724.291141039" watchObservedRunningTime="2026-10-30 18:59:21.585432216 +0000 UTC m=+3724.291617572" Oct 30 18:59:24.188846 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:24.188808 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:25.182994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:25.182957 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:25.411124 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:25.411100 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:30.408232 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:30.408196 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:30.416197 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:30.416172 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:30.591359 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:30.591335 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:30.652703 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:30.652672 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:30.950103 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:30.950070 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:31.175092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:31.175064 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:32.287555 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:32.287523 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:32.288000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:32.287759 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="31edd20e-95e3-4c28-9b56-56302ee95305" containerName="otc-container" containerID="cri-o://00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06" gracePeriod=30 Oct 30 18:59:32.558560 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:32.558446 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:32.863073 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:32.863033 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:33.540587 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.540563 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:33.605622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.605440 6966 generic.go:356] "Generic (PLEG): container finished" podID="31edd20e-95e3-4c28-9b56-56302ee95305" containerID="00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06" exitCode=0 Oct 30 18:59:33.605622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.605503 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"31edd20e-95e3-4c28-9b56-56302ee95305","Type":"ContainerDied","Data":"00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06"} Oct 30 18:59:33.605622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.605532 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:33.605622 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.605582 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"31edd20e-95e3-4c28-9b56-56302ee95305","Type":"ContainerDied","Data":"f7f9212dc4d53e0b510b789e0830949b1d1b1a7c1d160795b974764f7dba318d"} Oct 30 18:59:33.606024 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.605669 6966 scope.go:122] "RemoveContainer" containerID="00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06" Oct 30 18:59:33.619212 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.619192 6966 scope.go:122] "RemoveContainer" containerID="00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06" Oct 30 18:59:33.619616 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:33.619574 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06\": container with ID starting with 00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06 not found: ID does not exist" containerID="00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06" Oct 30 18:59:33.619724 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.619630 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06"} err="failed to get container status \"00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06\": rpc error: code = NotFound desc = could not find container \"00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06\": container with ID starting with 00e5e0b516d93f41b061aec417e8363d7a57be4bd70f87828e49da640bd74e06 not found: ID does not exist" Oct 30 18:59:33.689381 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.689356 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs\") pod \"31edd20e-95e3-4c28-9b56-56302ee95305\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " Oct 30 18:59:33.689539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.689424 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert\") pod \"31edd20e-95e3-4c28-9b56-56302ee95305\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " Oct 30 18:59:33.689539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.689477 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs\") pod \"31edd20e-95e3-4c28-9b56-56302ee95305\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " Oct 30 18:59:33.689539 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.689516 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n\" (UniqueName: \"kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n\") pod \"31edd20e-95e3-4c28-9b56-56302ee95305\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " Oct 30 18:59:33.689743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.689618 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal\" (UniqueName: \"kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal\") pod \"31edd20e-95e3-4c28-9b56-56302ee95305\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " Oct 30 18:59:33.689743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.689687 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\") pod \"31edd20e-95e3-4c28-9b56-56302ee95305\" (UID: \"31edd20e-95e3-4c28-9b56-56302ee95305\") " Oct 30 18:59:33.690061 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.690038 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal" pod "31edd20e-95e3-4c28-9b56-56302ee95305" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:33.691720 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.691699 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs" pod "31edd20e-95e3-4c28-9b56-56302ee95305" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:33.691999 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.691974 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs" pod "31edd20e-95e3-4c28-9b56-56302ee95305" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:33.692172 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.692146 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n" pod "31edd20e-95e3-4c28-9b56-56302ee95305" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305"). InnerVolumeSpecName "kube-api-access-8xg9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:33.692269 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.692187 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs" pod "31edd20e-95e3-4c28-9b56-56302ee95305" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:33.692400 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.692384 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert" pod "31edd20e-95e3-4c28-9b56-56302ee95305" (UID: "31edd20e-95e3-4c28-9b56-56302ee95305"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:33.790991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.790960 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.790991 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.790988 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.791182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.790998 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.791182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.791008 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xg9n\" (UniqueName: \"kubernetes.io/projected/31edd20e-95e3-4c28-9b56-56302ee95305-kube-api-access-8xg9n\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.791182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.791017 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/31edd20e-95e3-4c28-9b56-56302ee95305-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.791182 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.791025 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/31edd20e-95e3-4c28-9b56-56302ee95305-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:33.948824 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.948771 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:33.960402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.960369 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:33.998830 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.998799 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:33.999335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.999311 6966 cpu_manager.go:414] "containerMap: removing container" podUID="31edd20e-95e3-4c28-9b56-56302ee95305" containerName="otc-container" Oct 30 18:59:33.999434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.999362 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31edd20e-95e3-4c28-9b56-56302ee95305" containerName="otc-container" Oct 30 18:59:33.999597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:33.999574 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="31edd20e-95e3-4c28-9b56-56302ee95305" containerName="otc-container" Oct 30 18:59:34.004543 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.004519 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.018882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.018861 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:59:34.018882 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.018870 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:59:34.019041 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.018875 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:59:34.019122 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.019093 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-bhcr4\"" Oct 30 18:59:34.019168 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.019134 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:59:34.019416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.019403 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:59:34.031840 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.031813 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:34.194778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.194695 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtqdx\" (UniqueName: \"kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.194778 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.194731 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.195021 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.194922 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.195021 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.194980 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.195138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.195033 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.195138 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.195066 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.296337 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.296305 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.296535 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.296484 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.296625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.296539 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.296625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.296564 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.296625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.296589 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtqdx\" (UniqueName: \"kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.296625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.296613 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.297296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.297266 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.299268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.299249 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.299416 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.299395 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.299548 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.299534 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.299740 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.299724 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.319733 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.319715 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtqdx\" (UniqueName: \"kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx\") pod \"data-science-collector-collector-1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.619365 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.619335 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:34.926108 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.926027 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:34.929734 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:34.929704 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c8b4701_f897_4c90_8df3_d5da23788ae1.slice/crio-641ec24fdaee82352b92ed4b467c272a5689144e9b3b24f1ba36231e4d75ab71 WatchSource:0}: Error finding container 641ec24fdaee82352b92ed4b467c272a5689144e9b3b24f1ba36231e4d75ab71: Status 404 returned error can't find the container with id 641ec24fdaee82352b92ed4b467c272a5689144e9b3b24f1ba36231e4d75ab71 Oct 30 18:59:34.932214 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.932178 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:59:34.932296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:34.932248 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:59:35.411310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:35.411275 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31edd20e-95e3-4c28-9b56-56302ee95305" path="/var/lib/kubelet/pods/31edd20e-95e3-4c28-9b56-56302ee95305/volumes" Oct 30 18:59:35.618569 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:35.618529 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c8b4701-f897-4c90-8df3-d5da23788ae1","Type":"ContainerStarted","Data":"b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda"} Oct 30 18:59:35.618713 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:35.618586 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c8b4701-f897-4c90-8df3-d5da23788ae1","Type":"ContainerStarted","Data":"641ec24fdaee82352b92ed4b467c272a5689144e9b3b24f1ba36231e4d75ab71"} Oct 30 18:59:39.005915 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:39.005874 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:39.945654 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:39.945601 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:40.183067 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:40.183034 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:47.199129 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:47.199085 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 18:59:47.846795 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:47.846758 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:48.077264 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:48.077235 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.332099 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:49.332047 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=16.332035836 podStartE2EDuration="16.332035836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:35.671882421 +0000 UTC m=+3738.378067791" watchObservedRunningTime="2026-10-30 18:59:49.332035836 +0000 UTC m=+3752.038221193" Oct 30 18:59:49.333929 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:49.333901 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:49.334279 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:49.334252 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2c8b4701-f897-4c90-8df3-d5da23788ae1" containerName="otc-container" containerID="cri-o://b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda" gracePeriod=30 Oct 30 18:59:49.614508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:49.614396 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:49.918612 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:49.918528 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:50.579040 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.579015 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.645203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.645135 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs\") pod \"2c8b4701-f897-4c90-8df3-d5da23788ae1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " Oct 30 18:59:50.645203 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.645175 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs\") pod \"2c8b4701-f897-4c90-8df3-d5da23788ae1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " Oct 30 18:59:50.645438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.645233 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx\" (UniqueName: \"kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx\") pod \"2c8b4701-f897-4c90-8df3-d5da23788ae1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " Oct 30 18:59:50.645438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.645264 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs\") pod \"2c8b4701-f897-4c90-8df3-d5da23788ae1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " Oct 30 18:59:50.645438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.645307 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal\") pod \"2c8b4701-f897-4c90-8df3-d5da23788ae1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " Oct 30 18:59:50.645438 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.645355 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert\") pod \"2c8b4701-f897-4c90-8df3-d5da23788ae1\" (UID: \"2c8b4701-f897-4c90-8df3-d5da23788ae1\") " Oct 30 18:59:50.646092 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.646023 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal" pod "2c8b4701-f897-4c90-8df3-d5da23788ae1" (UID: "2c8b4701-f897-4c90-8df3-d5da23788ae1"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:50.647908 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.647878 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert" pod "2c8b4701-f897-4c90-8df3-d5da23788ae1" (UID: "2c8b4701-f897-4c90-8df3-d5da23788ae1"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:50.648036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.647986 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx" pod "2c8b4701-f897-4c90-8df3-d5da23788ae1" (UID: "2c8b4701-f897-4c90-8df3-d5da23788ae1"). InnerVolumeSpecName "kube-api-access-wtqdx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:50.648036 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.648007 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs" pod "2c8b4701-f897-4c90-8df3-d5da23788ae1" (UID: "2c8b4701-f897-4c90-8df3-d5da23788ae1"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:50.648117 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.648045 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs" pod "2c8b4701-f897-4c90-8df3-d5da23788ae1" (UID: "2c8b4701-f897-4c90-8df3-d5da23788ae1"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:50.649177 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.649154 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs" pod "2c8b4701-f897-4c90-8df3-d5da23788ae1" (UID: "2c8b4701-f897-4c90-8df3-d5da23788ae1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:50.699667 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.699639 6966 generic.go:356] "Generic (PLEG): container finished" podID="2c8b4701-f897-4c90-8df3-d5da23788ae1" containerID="b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda" exitCode=0 Oct 30 18:59:50.699766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.699686 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.699766 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.699720 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c8b4701-f897-4c90-8df3-d5da23788ae1","Type":"ContainerDied","Data":"b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda"} Oct 30 18:59:50.699845 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.699772 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2c8b4701-f897-4c90-8df3-d5da23788ae1","Type":"ContainerDied","Data":"641ec24fdaee82352b92ed4b467c272a5689144e9b3b24f1ba36231e4d75ab71"} Oct 30 18:59:50.699845 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.699798 6966 scope.go:122] "RemoveContainer" containerID="b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda" Oct 30 18:59:50.709918 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.709895 6966 scope.go:122] "RemoveContainer" containerID="b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda" Oct 30 18:59:50.710183 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:50.710165 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda\": container with ID starting with b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda not found: ID does not exist" containerID="b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda" Oct 30 18:59:50.710233 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.710205 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda"} err="failed to get container status \"b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda\": rpc error: code = NotFound desc = could not find container \"b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda\": container with ID starting with b80f71bd3c170e8c64d25fedefeaae69b6960a12190655a07814777e61837cda not found: ID does not exist" Oct 30 18:59:50.746268 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.746247 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtqdx\" (UniqueName: \"kubernetes.io/projected/2c8b4701-f897-4c90-8df3-d5da23788ae1-kube-api-access-wtqdx\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:50.746393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.746311 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:50.746393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.746329 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2c8b4701-f897-4c90-8df3-d5da23788ae1-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:50.746393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.746348 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:50.746393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.746366 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:50.746393 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.746379 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2c8b4701-f897-4c90-8df3-d5da23788ae1-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:50.747859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.747807 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:50.752910 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.752883 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:50.789727 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.789696 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:50.790235 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.790217 6966 cpu_manager.go:414] "containerMap: removing container" podUID="2c8b4701-f897-4c90-8df3-d5da23788ae1" containerName="otc-container" Oct 30 18:59:50.790341 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.790236 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c8b4701-f897-4c90-8df3-d5da23788ae1" containerName="otc-container" Oct 30 18:59:50.790453 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.790370 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c8b4701-f897-4c90-8df3-d5da23788ae1" containerName="otc-container" Oct 30 18:59:50.803735 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.803710 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.815865 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.815832 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:50.817836 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.817804 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 18:59:50.817932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.817893 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-bhcr4\"" Oct 30 18:59:50.817932 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.817902 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 18:59:50.818064 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.817971 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 18:59:50.818135 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.818072 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 18:59:50.818325 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.818306 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 18:59:50.847624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.847599 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.847729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.847638 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.847729 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.847688 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.847854 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.847759 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwg6n\" (UniqueName: \"kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.847914 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.847899 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.847950 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.847932 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.948684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.948621 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.948684 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.948658 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.948877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.948688 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.948877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.948710 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.948877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.948741 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.948877 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.948773 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwg6n\" (UniqueName: \"kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.949434 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.949408 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.951721 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.951693 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.951823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.951743 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.951903 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.951885 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.952066 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.952051 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:50.970820 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:50.970798 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwg6n\" (UniqueName: \"kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n\") pod \"data-science-collector-collector-1\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:51.117530 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.117500 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:51.415479 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.415419 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c8b4701-f897-4c90-8df3-d5da23788ae1" path="/var/lib/kubelet/pods/2c8b4701-f897-4c90-8df3-d5da23788ae1/volumes" Oct 30 18:59:51.419596 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.419569 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:51.425710 ip-10-0-136-241 kubenswrapper[6966]: W1030 18:59:51.425670 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e6daeda_353b_4f4d_81bf_b47a8cc489a3.slice/crio-fd22dbf86d3e402aeff5b65be8e88f064f5381007eb693f58214b49406acd9b1 WatchSource:0}: Error finding container fd22dbf86d3e402aeff5b65be8e88f064f5381007eb693f58214b49406acd9b1: Status 404 returned error can't find the container with id fd22dbf86d3e402aeff5b65be8e88f064f5381007eb693f58214b49406acd9b1 Oct 30 18:59:51.428237 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.428201 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:59:51.428346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.428263 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 18:59:51.707597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.707503 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e6daeda-353b-4f4d-81bf-b47a8cc489a3","Type":"ContainerStarted","Data":"7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd"} Oct 30 18:59:51.707597 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.707563 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e6daeda-353b-4f4d-81bf-b47a8cc489a3","Type":"ContainerStarted","Data":"fd22dbf86d3e402aeff5b65be8e88f064f5381007eb693f58214b49406acd9b1"} Oct 30 18:59:51.749335 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:51.749266 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.749248525 podStartE2EDuration="1.749248525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:59:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:59:51.747100721 +0000 UTC m=+3754.453286070" watchObservedRunningTime="2026-10-30 18:59:51.749248525 +0000 UTC m=+3754.455433883" Oct 30 18:59:58.104951 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.104908 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn"] Oct 30 18:59:58.105523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.105216 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="kube-rbac-proxy" containerID="cri-o://43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" gracePeriod=30 Oct 30 18:59:58.105523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.105400 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="prom-label-proxy" containerID="cri-o://b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" gracePeriod=30 Oct 30 18:59:58.105523 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.105438 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="method-gate" containerID="cri-o://15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" gracePeriod=30 Oct 30 18:59:58.120825 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.120799 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx"] Oct 30 18:59:58.121021 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.121000 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" podUID="1742a849-4ba4-46c9-8df3-be18aac04c20" containerName="korrel8r" containerID="cri-o://52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665" gracePeriod=30 Oct 30 18:59:58.364833 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.364775 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:59:58.386823 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.386797 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:59:58.415942 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.415897 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\" (UniqueName: \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\") pod \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " Oct 30 18:59:58.416153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.416006 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2\" (UniqueName: \"kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2\") pod \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " Oct 30 18:59:58.416153 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.416043 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config\" (UniqueName: \"kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config\") pod \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\" (UID: \"0decf7ad-ae2a-4cf2-b429-06e105f0100e\") " Oct 30 18:59:58.417642 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.416907 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config" pod "0decf7ad-ae2a-4cf2-b429-06e105f0100e" (UID: "0decf7ad-ae2a-4cf2-b429-06e105f0100e"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.420941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.420873 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls" pod "0decf7ad-ae2a-4cf2-b429-06e105f0100e" (UID: "0decf7ad-ae2a-4cf2-b429-06e105f0100e"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:58.420941 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.420903 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2" pod "0decf7ad-ae2a-4cf2-b429-06e105f0100e" (UID: "0decf7ad-ae2a-4cf2-b429-06e105f0100e"). InnerVolumeSpecName "kube-api-access-drls2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.517837 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.517808 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config\" (UniqueName: \"kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config\") pod \"1742a849-4ba4-46c9-8df3-be18aac04c20\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " Oct 30 18:59:58.518000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.517908 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2\" (UniqueName: \"kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2\") pod \"1742a849-4ba4-46c9-8df3-be18aac04c20\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " Oct 30 18:59:58.518000 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.517949 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls\" (UniqueName: \"kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls\") pod \"1742a849-4ba4-46c9-8df3-be18aac04c20\" (UID: \"1742a849-4ba4-46c9-8df3-be18aac04c20\") " Oct 30 18:59:58.518318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.518222 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0decf7ad-ae2a-4cf2-b429-06e105f0100e-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.518318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.518247 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drls2\" (UniqueName: \"kubernetes.io/projected/0decf7ad-ae2a-4cf2-b429-06e105f0100e-kube-api-access-drls2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.518318 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.518261 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0decf7ad-ae2a-4cf2-b429-06e105f0100e-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.518625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.518485 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config" pod "1742a849-4ba4-46c9-8df3-be18aac04c20" (UID: "1742a849-4ba4-46c9-8df3-be18aac04c20"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:58.520798 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.520772 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls" pod "1742a849-4ba4-46c9-8df3-be18aac04c20" (UID: "1742a849-4ba4-46c9-8df3-be18aac04c20"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:58.520894 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.520779 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2" pod "1742a849-4ba4-46c9-8df3-be18aac04c20" (UID: "1742a849-4ba4-46c9-8df3-be18aac04c20"). InnerVolumeSpecName "kube-api-access-8jtj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:58.618145 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.618073 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:58.619006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.618971 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jtj2\" (UniqueName: \"kubernetes.io/projected/1742a849-4ba4-46c9-8df3-be18aac04c20-kube-api-access-8jtj2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.619006 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.619002 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/1742a849-4ba4-46c9-8df3-be18aac04c20-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.619184 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.619014 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1742a849-4ba4-46c9-8df3-be18aac04c20-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:58.650484 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.650437 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:59:58.651150 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.651115 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="prometheus" containerID="cri-o://d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" gracePeriod=600 Oct 30 18:59:58.651261 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.651202 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="config-reloader" containerID="cri-o://6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" gracePeriod=600 Oct 30 18:59:58.651397 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.651344 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="thanos-sidecar" containerID="cri-o://02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" gracePeriod=600 Oct 30 18:59:58.663432 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.663404 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 18:59:58.663782 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.663746 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="alertmanager" containerID="cri-o://69879deb2f51f1e395cb6c2b7e4f50e02ed1637e7e19b8a9073725648b2bd26b" gracePeriod=120 Oct 30 18:59:58.664144 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.663918 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="config-reloader" containerID="cri-o://37def417bb0953c7e02bfbb856157cca59110989f6fba04ecd62b53b164f6be8" gracePeriod=120 Oct 30 18:59:58.749274 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749244 6966 generic.go:356] "Generic (PLEG): container finished" podID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerID="b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" exitCode=0 Oct 30 18:59:58.749274 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749268 6966 generic.go:356] "Generic (PLEG): container finished" podID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerID="15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" exitCode=0 Oct 30 18:59:58.749274 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749278 6966 generic.go:356] "Generic (PLEG): container finished" podID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerID="43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" exitCode=0 Oct 30 18:59:58.749594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749316 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" Oct 30 18:59:58.749594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749325 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerDied","Data":"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc"} Oct 30 18:59:58.749594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749418 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerDied","Data":"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97"} Oct 30 18:59:58.749594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749480 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerDied","Data":"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76"} Oct 30 18:59:58.749594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749483 6966 scope.go:122] "RemoveContainer" containerID="b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" Oct 30 18:59:58.749594 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.749520 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn" event={"ID":"0decf7ad-ae2a-4cf2-b429-06e105f0100e","Type":"ContainerDied","Data":"07391dd728f3618fc74f471974d5c23c4a9cc003e785cfc846c6983168fe7ed1"} Oct 30 18:59:58.750867 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.750849 6966 generic.go:356] "Generic (PLEG): container finished" podID="1742a849-4ba4-46c9-8df3-be18aac04c20" containerID="52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665" exitCode=0 Oct 30 18:59:58.750955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.750878 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" event={"ID":"1742a849-4ba4-46c9-8df3-be18aac04c20","Type":"ContainerDied","Data":"52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665"} Oct 30 18:59:58.750955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.750908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" event={"ID":"1742a849-4ba4-46c9-8df3-be18aac04c20","Type":"ContainerDied","Data":"4385d904e8f529e2ca4834a7c11f37696974bc9cf98cc73644625370b51efdc5"} Oct 30 18:59:58.750955 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.750921 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx" Oct 30 18:59:58.776442 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.776417 6966 scope.go:122] "RemoveContainer" containerID="15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" Oct 30 18:59:58.785515 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.785493 6966 scope.go:122] "RemoveContainer" containerID="43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" Oct 30 18:59:58.793001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.792986 6966 scope.go:122] "RemoveContainer" containerID="b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" Oct 30 18:59:58.793289 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:58.793264 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": container with ID starting with b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc not found: ID does not exist" containerID="b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" Oct 30 18:59:58.793332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.793299 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc"} err="failed to get container status \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": rpc error: code = NotFound desc = could not find container \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": container with ID starting with b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc not found: ID does not exist" Oct 30 18:59:58.793332 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.793319 6966 scope.go:122] "RemoveContainer" containerID="15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" Oct 30 18:59:58.793654 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:58.793634 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": container with ID starting with 15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97 not found: ID does not exist" containerID="15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" Oct 30 18:59:58.793717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.793660 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97"} err="failed to get container status \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": rpc error: code = NotFound desc = could not find container \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": container with ID starting with 15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97 not found: ID does not exist" Oct 30 18:59:58.793717 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.793673 6966 scope.go:122] "RemoveContainer" containerID="43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" Oct 30 18:59:58.793900 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:58.793882 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": container with ID starting with 43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76 not found: ID does not exist" containerID="43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" Oct 30 18:59:58.793938 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.793905 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76"} err="failed to get container status \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": rpc error: code = NotFound desc = could not find container \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": container with ID starting with 43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76 not found: ID does not exist" Oct 30 18:59:58.793938 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.793919 6966 scope.go:122] "RemoveContainer" containerID="b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" Oct 30 18:59:58.794094 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794078 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc"} err="failed to get container status \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": rpc error: code = NotFound desc = could not find container \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": container with ID starting with b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc not found: ID does not exist" Oct 30 18:59:58.794134 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794095 6966 scope.go:122] "RemoveContainer" containerID="15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" Oct 30 18:59:58.794281 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794264 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97"} err="failed to get container status \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": rpc error: code = NotFound desc = could not find container \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": container with ID starting with 15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97 not found: ID does not exist" Oct 30 18:59:58.794326 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794281 6966 scope.go:122] "RemoveContainer" containerID="43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" Oct 30 18:59:58.794516 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794493 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76"} err="failed to get container status \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": rpc error: code = NotFound desc = could not find container \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": container with ID starting with 43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76 not found: ID does not exist" Oct 30 18:59:58.794600 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794518 6966 scope.go:122] "RemoveContainer" containerID="b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc" Oct 30 18:59:58.794796 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794777 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc"} err="failed to get container status \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": rpc error: code = NotFound desc = could not find container \"b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc\": container with ID starting with b3e6d87c4a5d7b80081f0f2895c8f254960227e1afe0c552b41cb797b3e3ddfc not found: ID does not exist" Oct 30 18:59:58.794847 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794797 6966 scope.go:122] "RemoveContainer" containerID="15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97" Oct 30 18:59:58.795001 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.794986 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97"} err="failed to get container status \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": rpc error: code = NotFound desc = could not find container \"15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97\": container with ID starting with 15218411ec421df478ac68e7a267baebaab3da079729a9d993ead747a87a6c97 not found: ID does not exist" Oct 30 18:59:58.795050 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.795000 6966 scope.go:122] "RemoveContainer" containerID="43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76" Oct 30 18:59:58.795176 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.795162 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76"} err="failed to get container status \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": rpc error: code = NotFound desc = could not find container \"43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76\": container with ID starting with 43bec025c9611da0780f51e7b9a9bd70d72215df634938a2b2406645794a1f76 not found: ID does not exist" Oct 30 18:59:58.795221 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.795176 6966 scope.go:122] "RemoveContainer" containerID="52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665" Oct 30 18:59:58.802668 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.802651 6966 scope.go:122] "RemoveContainer" containerID="52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665" Oct 30 18:59:58.802922 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:58.802905 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665\": container with ID starting with 52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665 not found: ID does not exist" containerID="52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665" Oct 30 18:59:58.802970 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.802926 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665"} err="failed to get container status \"52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665\": rpc error: code = NotFound desc = could not find container \"52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665\": container with ID starting with 52b60fa5dd48cd6a51be6298c89d0045efb8a4f04234c7ea1588be4432391665 not found: ID does not exist" Oct 30 18:59:58.821706 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.821680 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx"] Oct 30 18:59:58.832928 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.832908 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-ljsvx"] Oct 30 18:59:58.879919 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.879870 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn"] Oct 30 18:59:58.887262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.887234 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-mr9nn"] Oct 30 18:59:58.894757 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:58.894728 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.150624 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.150561 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.364743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.361180 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:59.364743 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.361471 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="2e6daeda-353b-4f4d-81bf-b47a8cc489a3" containerName="otc-container" containerID="cri-o://7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd" gracePeriod=30 Oct 30 18:59:59.416296 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.416228 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" path="/var/lib/kubelet/pods/0decf7ad-ae2a-4cf2-b429-06e105f0100e/volumes" Oct 30 18:59:59.416988 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.416958 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1742a849-4ba4-46c9-8df3-be18aac04c20" path="/var/lib/kubelet/pods/1742a849-4ba4-46c9-8df3-be18aac04c20/volumes" Oct 30 18:59:59.460472 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.460428 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.491095 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.491068 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:59.610398 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.610374 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:59.630205 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630184 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630256 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630329 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630295 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630335 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630366 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630476 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630411 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630476 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630634 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630565 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630656 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630744 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630692 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630744 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630780 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.630853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.630829 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file\") pod \"7260339e-45a6-495a-980f-1e643e959bdc\" (UID: \"7260339e-45a6-495a-980f-1e643e959bdc\") " Oct 30 18:59:59.631710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.631454 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.631710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.631553 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.631710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.631661 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.633436 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.633400 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.633576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.633432 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.633576 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.633568 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.633886 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.633862 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "kube-api-access-4gpd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.633983 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.633888 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.634189 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.634148 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.634262 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.634230 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:59:59.635309 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.635287 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.636975 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.636954 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 18:59:59.646491 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.646446 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config" pod "7260339e-45a6-495a-980f-1e643e959bdc" (UID: "7260339e-45a6-495a-980f-1e643e959bdc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.698962 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.698885 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 18:59:59.732133 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732108 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert\") pod \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " Oct 30 18:59:59.732259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732143 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs\") pod \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " Oct 30 18:59:59.732259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732194 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs\") pod \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " Oct 30 18:59:59.732259 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732230 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs\") pod \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " Oct 30 18:59:59.732396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732270 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal\") pod \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " Oct 30 18:59:59.732396 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732321 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n\" (UniqueName: \"kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n\") pod \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\" (UID: \"2e6daeda-353b-4f4d-81bf-b47a8cc489a3\") " Oct 30 18:59:59.732633 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732613 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732639 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732658 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732678 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpd4\" (UniqueName: \"kubernetes.io/projected/7260339e-45a6-495a-980f-1e643e959bdc-kube-api-access-4gpd4\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732695 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732710 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732709 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7260339e-45a6-495a-980f-1e643e959bdc-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732723 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732736 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732751 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732764 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732688 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal" pod "2e6daeda-353b-4f4d-81bf-b47a8cc489a3" (UID: "2e6daeda-353b-4f4d-81bf-b47a8cc489a3"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732792 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732807 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/7260339e-45a6-495a-980f-1e643e959bdc-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.732935 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.732824 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/7260339e-45a6-495a-980f-1e643e959bdc-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.734967 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.734945 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs" pod "2e6daeda-353b-4f4d-81bf-b47a8cc489a3" (UID: "2e6daeda-353b-4f4d-81bf-b47a8cc489a3"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.735110 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.735086 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert" pod "2e6daeda-353b-4f4d-81bf-b47a8cc489a3" (UID: "2e6daeda-353b-4f4d-81bf-b47a8cc489a3"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.735174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.735096 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs" pod "2e6daeda-353b-4f4d-81bf-b47a8cc489a3" (UID: "2e6daeda-353b-4f4d-81bf-b47a8cc489a3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.735174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.735122 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n" pod "2e6daeda-353b-4f4d-81bf-b47a8cc489a3" (UID: "2e6daeda-353b-4f4d-81bf-b47a8cc489a3"). InnerVolumeSpecName "kube-api-access-lwg6n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:59:59.735174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.735147 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs" pod "2e6daeda-353b-4f4d-81bf-b47a8cc489a3" (UID: "2e6daeda-353b-4f4d-81bf-b47a8cc489a3"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:59:59.745258 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.745244 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 18:59:59.758681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758642 6966 generic.go:356] "Generic (PLEG): container finished" podID="7260339e-45a6-495a-980f-1e643e959bdc" containerID="02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" exitCode=0 Oct 30 18:59:59.758681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758665 6966 generic.go:356] "Generic (PLEG): container finished" podID="7260339e-45a6-495a-980f-1e643e959bdc" containerID="6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" exitCode=0 Oct 30 18:59:59.758681 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758674 6966 generic.go:356] "Generic (PLEG): container finished" podID="7260339e-45a6-495a-980f-1e643e959bdc" containerID="d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" exitCode=0 Oct 30 18:59:59.758901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758681 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerDied","Data":"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9"} Oct 30 18:59:59.758901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758752 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 18:59:59.758901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758767 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerDied","Data":"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847"} Oct 30 18:59:59.758901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758810 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerDied","Data":"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b"} Oct 30 18:59:59.758901 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758841 6966 scope.go:122] "RemoveContainer" containerID="02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" Oct 30 18:59:59.759174 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.758850 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"7260339e-45a6-495a-980f-1e643e959bdc","Type":"ContainerDied","Data":"1d55810be268f1142ca175ff3646203b8e46868e0da302829589fb23654f3893"} Oct 30 18:59:59.760683 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.760660 6966 generic.go:356] "Generic (PLEG): container finished" podID="2e6daeda-353b-4f4d-81bf-b47a8cc489a3" containerID="7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd" exitCode=0 Oct 30 18:59:59.760797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.760690 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e6daeda-353b-4f4d-81bf-b47a8cc489a3","Type":"ContainerDied","Data":"7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd"} Oct 30 18:59:59.760797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.760744 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"2e6daeda-353b-4f4d-81bf-b47a8cc489a3","Type":"ContainerDied","Data":"fd22dbf86d3e402aeff5b65be8e88f064f5381007eb693f58214b49406acd9b1"} Oct 30 18:59:59.760797 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.760701 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 18:59:59.770346 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.770331 6966 scope.go:122] "RemoveContainer" containerID="6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" Oct 30 18:59:59.770625 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.770601 6966 generic.go:356] "Generic (PLEG): container finished" podID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerID="37def417bb0953c7e02bfbb856157cca59110989f6fba04ecd62b53b164f6be8" exitCode=0 Oct 30 18:59:59.770853 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.770680 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerDied","Data":"37def417bb0953c7e02bfbb856157cca59110989f6fba04ecd62b53b164f6be8"} Oct 30 18:59:59.781712 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.781688 6966 scope.go:122] "RemoveContainer" containerID="d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" Oct 30 18:59:59.794508 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.792712 6966 scope.go:122] "RemoveContainer" containerID="6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928" Oct 30 18:59:59.803310 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.803287 6966 scope.go:122] "RemoveContainer" containerID="02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" Oct 30 18:59:59.803694 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:59.803673 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": container with ID starting with 02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9 not found: ID does not exist" containerID="02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" Oct 30 18:59:59.803759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.803699 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9"} err="failed to get container status \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": rpc error: code = NotFound desc = could not find container \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": container with ID starting with 02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9 not found: ID does not exist" Oct 30 18:59:59.803759 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.803717 6966 scope.go:122] "RemoveContainer" containerID="6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" Oct 30 18:59:59.804022 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:59.804002 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": container with ID starting with 6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847 not found: ID does not exist" containerID="6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" Oct 30 18:59:59.804111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804033 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847"} err="failed to get container status \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": rpc error: code = NotFound desc = could not find container \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": container with ID starting with 6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847 not found: ID does not exist" Oct 30 18:59:59.804111 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804053 6966 scope.go:122] "RemoveContainer" containerID="d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" Oct 30 18:59:59.804348 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:59.804321 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": container with ID starting with d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b not found: ID does not exist" containerID="d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" Oct 30 18:59:59.804439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804350 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b"} err="failed to get container status \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": rpc error: code = NotFound desc = could not find container \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": container with ID starting with d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b not found: ID does not exist" Oct 30 18:59:59.804439 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804374 6966 scope.go:122] "RemoveContainer" containerID="6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928" Oct 30 18:59:59.804657 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:59.804641 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": container with ID starting with 6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928 not found: ID does not exist" containerID="6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928" Oct 30 18:59:59.804699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804660 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928"} err="failed to get container status \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": rpc error: code = NotFound desc = could not find container \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": container with ID starting with 6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928 not found: ID does not exist" Oct 30 18:59:59.804699 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804671 6966 scope.go:122] "RemoveContainer" containerID="02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" Oct 30 18:59:59.804859 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804842 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9"} err="failed to get container status \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": rpc error: code = NotFound desc = could not find container \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": container with ID starting with 02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9 not found: ID does not exist" Oct 30 18:59:59.804898 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.804860 6966 scope.go:122] "RemoveContainer" containerID="6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" Oct 30 18:59:59.805017 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805003 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847"} err="failed to get container status \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": rpc error: code = NotFound desc = could not find container \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": container with ID starting with 6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847 not found: ID does not exist" Oct 30 18:59:59.805056 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805018 6966 scope.go:122] "RemoveContainer" containerID="d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" Oct 30 18:59:59.805180 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805165 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b"} err="failed to get container status \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": rpc error: code = NotFound desc = could not find container \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": container with ID starting with d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b not found: ID does not exist" Oct 30 18:59:59.805215 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805180 6966 scope.go:122] "RemoveContainer" containerID="6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928" Oct 30 18:59:59.805395 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805380 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928"} err="failed to get container status \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": rpc error: code = NotFound desc = could not find container \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": container with ID starting with 6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928 not found: ID does not exist" Oct 30 18:59:59.805443 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805395 6966 scope.go:122] "RemoveContainer" containerID="02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9" Oct 30 18:59:59.805601 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805577 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9"} err="failed to get container status \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": rpc error: code = NotFound desc = could not find container \"02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9\": container with ID starting with 02d34605385936ea6cbfa65755ee0a8e3ca2a0600003656e02a0b9c65ef8aee9 not found: ID does not exist" Oct 30 18:59:59.805645 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805604 6966 scope.go:122] "RemoveContainer" containerID="6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847" Oct 30 18:59:59.805800 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805785 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847"} err="failed to get container status \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": rpc error: code = NotFound desc = could not find container \"6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847\": container with ID starting with 6ea2b1ee2a10c10893c1c446bd613f5817b99488b1e92e7f4acbb6dc9523b847 not found: ID does not exist" Oct 30 18:59:59.805847 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805800 6966 scope.go:122] "RemoveContainer" containerID="d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b" Oct 30 18:59:59.805958 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805938 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b"} err="failed to get container status \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": rpc error: code = NotFound desc = could not find container \"d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b\": container with ID starting with d70f00fe655d887e00412d01df6faaf77a42542234ac8dd9bfd7677306cbe09b not found: ID does not exist" Oct 30 18:59:59.805994 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.805960 6966 scope.go:122] "RemoveContainer" containerID="6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928" Oct 30 18:59:59.806123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.806103 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928"} err="failed to get container status \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": rpc error: code = NotFound desc = could not find container \"6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928\": container with ID starting with 6090f5f1551ea6b1981e66eac8b91fa4f2c3e8dbc0cafecaed253239ff63b928 not found: ID does not exist" Oct 30 18:59:59.806123 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.806117 6966 scope.go:122] "RemoveContainer" containerID="7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd" Oct 30 18:59:59.813776 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.813761 6966 scope.go:122] "RemoveContainer" containerID="7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd" Oct 30 18:59:59.814035 ip-10-0-136-241 kubenswrapper[6966]: E1030 18:59:59.814015 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd\": container with ID starting with 7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd not found: ID does not exist" containerID="7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd" Oct 30 18:59:59.814107 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.814039 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd"} err="failed to get container status \"7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd\": rpc error: code = NotFound desc = could not find container \"7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd\": container with ID starting with 7275426f02266b9456408c728b483a47358641a94b2a3b12f0594d4b76b489bd not found: ID does not exist" Oct 30 18:59:59.816402 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.816379 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:59:59.821157 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.821137 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 18:59:59.834038 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834015 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwg6n\" (UniqueName: \"kubernetes.io/projected/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-kube-api-access-lwg6n\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.834142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834041 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.834142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834053 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.834142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834062 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.834142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834072 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 18:59:59.834142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834081 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.834142 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.834089 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/2e6daeda-353b-4f4d-81bf-b47a8cc489a3-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 18:59:59.859493 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.859448 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 18:59:59.863838 ip-10-0-136-241 kubenswrapper[6966]: I1030 18:59:59.863818 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:00.389614 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:00.389370 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:00.489951 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:00.489908 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.165:9093/-/ready\": dial tcp 10.134.0.165:9093: connect: connection refused" Oct 30 19:00:00.603282 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:00.603251 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:01.412539 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:01.412499 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e6daeda-353b-4f4d-81bf-b47a8cc489a3" path="/var/lib/kubelet/pods/2e6daeda-353b-4f4d-81bf-b47a8cc489a3/volumes" Oct 30 19:00:01.413140 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:01.413117 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7260339e-45a6-495a-980f-1e643e959bdc" path="/var/lib/kubelet/pods/7260339e-45a6-495a-980f-1e643e959bdc/volumes" Oct 30 19:00:05.489070 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:05.489035 6966 prober.go:122] "Probe failed" probeType="Readiness" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="alertmanager" probeResult="failure" output="Get \"http://10.134.0.165:9093/-/ready\": dial tcp 10.134.0.165:9093: connect: connection refused" Oct 30 19:00:08.821161 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:08.821129 6966 generic.go:356] "Generic (PLEG): container finished" podID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerID="69879deb2f51f1e395cb6c2b7e4f50e02ed1637e7e19b8a9073725648b2bd26b" exitCode=0 Oct 30 19:00:08.821582 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:08.821196 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerDied","Data":"69879deb2f51f1e395cb6c2b7e4f50e02ed1637e7e19b8a9073725648b2bd26b"} Oct 30 19:00:08.914760 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:08.914738 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:09.006218 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006154 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006218 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006204 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006431 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006237 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006431 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006256 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006431 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006282 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006431 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006342 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006431 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006370 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config\") pod \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\" (UID: \"ea4c466c-2b85-4cdd-9933-a45429afd9b5\") " Oct 30 19:00:09.006752 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.006647 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:09.008615 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.008566 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "kube-api-access-qvxd8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:09.008926 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.008893 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:09.009044 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.008940 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:09.009212 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.009187 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:09.012865 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.012844 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:09.019453 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.019434 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config" pod "ea4c466c-2b85-4cdd-9933-a45429afd9b5" (UID: "ea4c466c-2b85-4cdd-9933-a45429afd9b5"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:09.106881 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106858 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.107002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106884 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.107002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106897 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.107002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106908 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.107002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106942 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.107002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106956 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qvxd8\" (UniqueName: \"kubernetes.io/projected/ea4c466c-2b85-4cdd-9933-a45429afd9b5-kube-api-access-qvxd8\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.107002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.106970 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ea4c466c-2b85-4cdd-9933-a45429afd9b5-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:09.828193 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.828151 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" event={"ID":"ea4c466c-2b85-4cdd-9933-a45429afd9b5","Type":"ContainerDied","Data":"9c5e8fb0d8ddf77ba31281ffb823419dd56625594fafc3088e6540807651a198"} Oct 30 19:00:09.828193 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.828196 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1" Oct 30 19:00:09.828663 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.828239 6966 scope.go:122] "RemoveContainer" containerID="37def417bb0953c7e02bfbb856157cca59110989f6fba04ecd62b53b164f6be8" Oct 30 19:00:09.836734 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.836715 6966 scope.go:122] "RemoveContainer" containerID="69879deb2f51f1e395cb6c2b7e4f50e02ed1637e7e19b8a9073725648b2bd26b" Oct 30 19:00:09.844185 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.844169 6966 scope.go:122] "RemoveContainer" containerID="19a7635b43f9b8dc3893383fb6d58d2e55b34874c52ed50cf095441a330afb97" Oct 30 19:00:09.937018 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.936943 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:00:09.963845 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:09.963819 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-1"] Oct 30 19:00:11.411750 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.411719 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" path="/var/lib/kubelet/pods/ea4c466c-2b85-4cdd-9933-a45429afd9b5/volumes" Oct 30 19:00:11.463280 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.463249 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:11.707198 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.707116 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:11.720503 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720475 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc"] Oct 30 19:00:11.720941 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720925 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="method-gate" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720943 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="method-gate" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720958 6966 cpu_manager.go:414] "containerMap: removing container" podUID="1742a849-4ba4-46c9-8df3-be18aac04c20" containerName="korrel8r" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720964 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1742a849-4ba4-46c9-8df3-be18aac04c20" containerName="korrel8r" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720972 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="thanos-sidecar" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720983 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="thanos-sidecar" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720994 6966 cpu_manager.go:414] "containerMap: removing container" podUID="2e6daeda-353b-4f4d-81bf-b47a8cc489a3" containerName="otc-container" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.720999 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e6daeda-353b-4f4d-81bf-b47a8cc489a3" containerName="otc-container" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721014 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="alertmanager" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721019 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="alertmanager" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721034 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="prom-label-proxy" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721039 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="prom-label-proxy" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721047 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="config-reloader" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721051 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="config-reloader" Oct 30 19:00:11.721060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721061 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="kube-rbac-proxy" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721066 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="kube-rbac-proxy" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721073 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="init-config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721077 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="init-config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721084 6966 cpu_manager.go:414] "containerMap: removing container" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721087 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721093 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="prometheus" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721100 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="prometheus" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721113 6966 cpu_manager.go:414] "containerMap: removing container" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="init-config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721117 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="init-config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721198 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="kube-rbac-proxy" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721208 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="config-reloader" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721215 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="prometheus" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721227 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="method-gate" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721232 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="1742a849-4ba4-46c9-8df3-be18aac04c20" containerName="korrel8r" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721238 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="thanos-sidecar" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721243 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e6daeda-353b-4f4d-81bf-b47a8cc489a3" containerName="otc-container" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721250 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea4c466c-2b85-4cdd-9933-a45429afd9b5" containerName="alertmanager" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721256 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0decf7ad-ae2a-4cf2-b429-06e105f0100e" containerName="prom-label-proxy" Oct 30 19:00:11.721599 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.721260 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="7260339e-45a6-495a-980f-1e643e959bdc" containerName="config-reloader" Oct 30 19:00:11.725120 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.725102 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.754337 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.754317 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc"] Oct 30 19:00:11.755162 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.755137 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-tls\"" Oct 30 19:00:11.755162 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.755144 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 19:00:11.755307 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.755169 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-config\"" Oct 30 19:00:11.755307 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.755252 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"kube-root-ca.crt\"" Oct 30 19:00:11.755307 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.755291 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-thanos-querier-proxy-dockercfg-l6zd5\"" Oct 30 19:00:11.757135 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.757119 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c"] Oct 30 19:00:11.759758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.759740 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.781446 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.781421 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c"] Oct 30 19:00:11.828022 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.827995 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmhlp\" (UniqueName: \"kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.828117 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.828029 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb69g\" (UniqueName: \"kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.828117 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.828048 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.828200 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.828162 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.828238 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.828209 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.828277 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.828262 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.855420 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.855401 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-tls\"" Oct 30 19:00:11.855420 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.855409 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-config\"" Oct 30 19:00:11.855570 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.855513 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"korrel8r-dockercfg-q9r6f\"" Oct 30 19:00:11.929446 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.929419 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmhlp\" (UniqueName: \"kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.929576 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.929451 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb69g\" (UniqueName: \"kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.929576 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.929489 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.929576 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.929512 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.929576 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.929549 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.929576 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.929567 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.929865 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:11.929661 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/korrel8r-tls: secret "korrel8r-tls" not found Oct 30 19:00:11.929865 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:11.929683 6966 secret.go:189] Couldn't get secret redhat-ods-monitoring/data-science-thanos-querier-proxy-tls: secret "data-science-thanos-querier-proxy-tls" not found Oct 30 19:00:11.929865 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:11.929727 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls podName:80238df0-4f32-4963-81ad-82f293141376 nodeName:}" failed. No retries permitted until 2026-10-30 19:00:12.42970626 +0000 UTC m=+3775.135891610 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls") pod "korrel8r-5cbcc89b8f-lgv8c" (UID: "80238df0-4f32-4963-81ad-82f293141376") : secret "korrel8r-tls" not found Oct 30 19:00:11.929865 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:11.929746 6966 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls podName:0b834912-5b9f-4a4a-aa48-1e2d48c11bec nodeName:}" failed. No retries permitted until 2026-10-30 19:00:12.429736734 +0000 UTC m=+3775.135922075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls" (UniqueName: "kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls") pod "data-science-thanos-querier-proxy-6998ff8b98-hh6qc" (UID: "0b834912-5b9f-4a4a-aa48-1e2d48c11bec") : secret "data-science-thanos-querier-proxy-tls" not found Oct 30 19:00:11.930224 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.930207 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.930348 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.930332 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:11.945757 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.945731 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:11.952356 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.952325 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb69g\" (UniqueName: \"kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:11.952544 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:11.952525 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmhlp\" (UniqueName: \"kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:12.174340 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.174307 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.396995 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.396963 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.434045 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.433972 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:12.434045 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.434013 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls\" (UniqueName: \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:12.436385 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.436360 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\") pod \"data-science-thanos-querier-proxy-6998ff8b98-hh6qc\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:12.436527 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.436491 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls\" (UniqueName: \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\") pod \"korrel8r-5cbcc89b8f-lgv8c\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:12.640197 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.640146 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.640197 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.640194 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:12.671657 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.671625 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:12.893796 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.893763 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:12.941277 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.941209 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc"] Oct 30 19:00:12.945524 ip-10-0-136-241 kubenswrapper[6966]: W1030 19:00:12.945497 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b834912_5b9f_4a4a_aa48_1e2d48c11bec.slice/crio-96373640c8d522ec84df1c5660b2051eaf2b507e082e37c53b9157e3bd23f0ae WatchSource:0}: Error finding container 96373640c8d522ec84df1c5660b2051eaf2b507e082e37c53b9157e3bd23f0ae: Status 404 returned error can't find the container with id 96373640c8d522ec84df1c5660b2051eaf2b507e082e37c53b9157e3bd23f0ae Oct 30 19:00:12.975140 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:12.975121 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c"] Oct 30 19:00:12.990544 ip-10-0-136-241 kubenswrapper[6966]: W1030 19:00:12.990515 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80238df0_4f32_4963_81ad_82f293141376.slice/crio-374cff1b9120ced4e98333a416620b0af5b596541d56a72d7108f61420dc9c56 WatchSource:0}: Error finding container 374cff1b9120ced4e98333a416620b0af5b596541d56a72d7108f61420dc9c56: Status 404 returned error can't find the container with id 374cff1b9120ced4e98333a416620b0af5b596541d56a72d7108f61420dc9c56 Oct 30 19:00:13.190676 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.190652 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:13.195751 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.195696 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.216310 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.216283 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:13.223178 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.223155 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-a0f9182a\"" Oct 30 19:00:13.251323 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.251296 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-monitoring-tls\"" Oct 30 19:00:13.251323 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.251326 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-dockercfg-npstw\"" Oct 30 19:00:13.251613 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.251384 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-collector-headless-tls\"" Oct 30 19:00:13.251613 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.251586 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-ta-client-cert\"" Oct 30 19:00:13.251720 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.251654 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-collector-tls\"" Oct 30 19:00:13.314996 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.314968 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:13.342169 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.342142 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbx8v\" (UniqueName: \"kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.342322 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.342187 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.342322 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.342271 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.342393 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.342344 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.342393 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.342371 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.342523 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.342439 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.443453 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.443425 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.443874 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.443501 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.443874 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.443521 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.443874 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.443538 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.443874 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.443561 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.443874 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.443699 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbx8v\" (UniqueName: \"kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.444325 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.444303 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.446023 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.445967 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.446120 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.446041 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.446120 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.446053 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.446207 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.446121 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.465941 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.465922 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbx8v\" (UniqueName: \"kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v\") pod \"data-science-collector-collector-1\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.510574 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.510553 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:13.535794 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.534355 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:13.812065 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.812037 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:13.817740 ip-10-0-136-241 kubenswrapper[6966]: W1030 19:00:13.817711 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb40bf11b_907a_451f_bd41_14cf7ec85544.slice/crio-024f7f1dfbaa7534de93218aed0736d722457f60744c15fc0ebaec36a1a5de2c WatchSource:0}: Error finding container 024f7f1dfbaa7534de93218aed0736d722457f60744c15fc0ebaec36a1a5de2c: Status 404 returned error can't find the container with id 024f7f1dfbaa7534de93218aed0736d722457f60744c15fc0ebaec36a1a5de2c Oct 30 19:00:13.819970 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.819939 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 19:00:13.820049 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.820003 6966 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236064Ki","pods":"250"} Oct 30 19:00:13.850665 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.850456 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" event={"ID":"80238df0-4f32-4963-81ad-82f293141376","Type":"ContainerStarted","Data":"766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480"} Oct 30 19:00:13.850665 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.850544 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:13.850665 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.850562 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" event={"ID":"80238df0-4f32-4963-81ad-82f293141376","Type":"ContainerStarted","Data":"374cff1b9120ced4e98333a416620b0af5b596541d56a72d7108f61420dc9c56"} Oct 30 19:00:13.852809 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.852775 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerStarted","Data":"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6"} Oct 30 19:00:13.852935 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.852836 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerStarted","Data":"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046"} Oct 30 19:00:13.852935 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.852872 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerStarted","Data":"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392"} Oct 30 19:00:13.852935 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.852908 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerStarted","Data":"96373640c8d522ec84df1c5660b2051eaf2b507e082e37c53b9157e3bd23f0ae"} Oct 30 19:00:13.854046 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.854027 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b40bf11b-907a-451f-bd41-14cf7ec85544","Type":"ContainerStarted","Data":"024f7f1dfbaa7534de93218aed0736d722457f60744c15fc0ebaec36a1a5de2c"} Oct 30 19:00:13.889028 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.888987 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" podStartSLOduration=2.888970413 podStartE2EDuration="2.888970413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:13.888534447 +0000 UTC m=+3776.594719805" watchObservedRunningTime="2026-10-30 19:00:13.888970413 +0000 UTC m=+3776.595155771" Oct 30 19:00:13.931009 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:13.930957 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" podStartSLOduration=2.930944103 podStartE2EDuration="2.930944103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:13.92962612 +0000 UTC m=+3776.635811477" watchObservedRunningTime="2026-10-30 19:00:13.930944103 +0000 UTC m=+3776.637129459" Oct 30 19:00:14.862002 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:14.861964 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b40bf11b-907a-451f-bd41-14cf7ec85544","Type":"ContainerStarted","Data":"2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf"} Oct 30 19:00:14.906283 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:14.906238 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/data-science-collector-collector-1" podStartSLOduration=1.9062260979999999 podStartE2EDuration="1.906226098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:14.905099352 +0000 UTC m=+3777.611284688" watchObservedRunningTime="2026-10-30 19:00:14.906226098 +0000 UTC m=+3777.612411454" Oct 30 19:00:17.199356 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:17.199315 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 19:00:20.460655 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:20.460617 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:20.699795 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:20.699756 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:20.924616 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:20.924581 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:21.008521 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.008187 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 19:00:21.014572 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.014543 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.028663 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.028639 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-tls-assets-0\"" Oct 30 19:00:21.034246 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.034222 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-web-config\"" Oct 30 19:00:21.034403 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.034269 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-generated\"" Oct 30 19:00:21.048775 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.048747 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 19:00:21.069489 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.069451 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 19:00:21.073783 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.073759 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.094274 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.094255 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-1\"" Oct 30 19:00:21.094379 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.094275 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-2\"" Oct 30 19:00:21.094379 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.094365 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-rulefiles-0\"" Oct 30 19:00:21.106222 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.106198 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 19:00:21.109188 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109155 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.109303 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109219 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.109349 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109296 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.109349 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109328 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.109430 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109410 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.109484 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109429 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qvnc\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.109529 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.109495 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.128883 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.128862 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-web-tls-ca\"" Oct 30 19:00:21.128883 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.128861 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-thanos-prometheus-http-client-file\"" Oct 30 19:00:21.128883 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.128864 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-web-config\"" Oct 30 19:00:21.129306 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.129281 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"alertmanager-data-science-monitoringstack-cluster-tls-config\"" Oct 30 19:00:21.129306 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.129292 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-prometheus-dockercfg-72b7s\"" Oct 30 19:00:21.129306 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.129282 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack\"" Oct 30 19:00:21.129519 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.129305 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-data-science-monitoringstack-tls-assets-0\"" Oct 30 19:00:21.129519 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.129382 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"data-science-monitoringstack-alertmanager-dockercfg-hg6c9\"" Oct 30 19:00:21.129519 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.129404 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"redhat-ods-monitoring\"/\"prometheus-operated-tls\"" Oct 30 19:00:21.166043 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.166014 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:21.210079 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210008 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.210079 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210045 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.210332 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210079 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210332 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210112 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.210332 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210179 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.210332 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210208 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qvnc\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.210332 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210240 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zm99\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210332 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210263 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210336 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.210758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210365 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210413 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210522 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210562 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.210758 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210677 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.211105 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210762 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.211105 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210808 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.211105 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210841 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.211105 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210896 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.211105 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.210949 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.211382 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.211142 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.212710 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.212674 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.212819 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.212710 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.212897 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.212879 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.213057 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.213042 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.213104 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.213070 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.233110 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.233088 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qvnc\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc\") pod \"alertmanager-data-science-monitoringstack-0\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.311850 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.311823 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.311941 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.311863 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.311941 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.311891 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.311941 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.311921 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312125 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.311957 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312125 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.311987 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312125 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312005 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312125 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312045 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312125 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312084 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312391 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312144 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zm99\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312391 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312177 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312391 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312209 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.312607 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.312508 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.313388 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.313070 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.313388 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.313323 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.314785 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.314755 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.314887 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.314862 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.314887 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.314870 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.314987 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.314905 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.315051 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.315034 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.315133 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.315108 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.315865 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.315848 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.316501 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.316485 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.333360 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.333340 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:21.336419 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.336400 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zm99\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:21.450258 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.450223 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:21.664886 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.664851 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 19:00:21.668944 ip-10-0-136-241 kubenswrapper[6966]: W1030 19:00:21.668922 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6695d7ff_9c2a_45e2_bbc2_24e4bd00fc35.slice/crio-55f175f00b33c5bb1020bbd2c35150b1e98b11024b6cfb2d300e98805f707793 WatchSource:0}: Error finding container 55f175f00b33c5bb1020bbd2c35150b1e98b11024b6cfb2d300e98805f707793: Status 404 returned error can't find the container with id 55f175f00b33c5bb1020bbd2c35150b1e98b11024b6cfb2d300e98805f707793 Oct 30 19:00:21.900772 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:21.900738 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerStarted","Data":"55f175f00b33c5bb1020bbd2c35150b1e98b11024b6cfb2d300e98805f707793"} Oct 30 19:00:22.906809 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:22.906774 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerStarted","Data":"039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1"} Oct 30 19:00:24.293076 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:24.293043 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:24.864473 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:24.864435 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:25.284198 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:25.284115 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:25.924850 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:25.924820 6966 generic.go:356] "Generic (PLEG): container finished" podID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerID="039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1" exitCode=0 Oct 30 19:00:25.925212 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:25.924897 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerDied","Data":"039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1"} Oct 30 19:00:26.660528 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.660490 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.689194 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.689132 6966 operation_generator.go:1464] "Controller attach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") device path: \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.761789 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.761763 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.761933 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.761873 6966 operation_generator.go:515] "MountVolume.WaitForAttach entering for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") DevicePath \"\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.775368 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.775341 6966 operation_generator.go:525] "MountVolume.WaitForAttach succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.878049 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.878024 6966 operation_generator.go:557] "MountVolume.MountDevice succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") device mount path \"/var/lib/kubelet/plugins/kubernetes.io/csi/ebs.csi.aws.com/3d0cf4484736f6d40d16a5331fa2b97b9730899f1120a88b3f09811f40fc91e0/globalmount\"" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.905841 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.905818 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"prometheus-data-science-monitoringstack-0\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:26.932227 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.932202 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerStarted","Data":"9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad"} Oct 30 19:00:26.932531 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.932257 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerStarted","Data":"2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6"} Oct 30 19:00:26.932531 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.932433 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:26.979159 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:26.979084 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podStartSLOduration=3.592794417 podStartE2EDuration="6.979072322s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.386277905s" isStatefulPod=false podCreationTimestamp="2026-10-30 19:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:26.97791289 +0000 UTC m=+3789.684098246" watchObservedRunningTime="2026-10-30 19:00:26.979072322 +0000 UTC m=+3789.685257675" Oct 30 19:00:27.094590 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:27.094568 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:27.448884 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:27.448851 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 19:00:27.456112 ip-10-0-136-241 kubenswrapper[6966]: W1030 19:00:27.456083 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cfe905f_2910_478c_9712_68ba0b12ae3f.slice/crio-ff474ea61d17d7682458638a617ec8d10e81cef318e7f9d071f77e2ff57c69d7 WatchSource:0}: Error finding container ff474ea61d17d7682458638a617ec8d10e81cef318e7f9d071f77e2ff57c69d7: Status 404 returned error can't find the container with id ff474ea61d17d7682458638a617ec8d10e81cef318e7f9d071f77e2ff57c69d7 Oct 30 19:00:27.938349 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:27.938318 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerStarted","Data":"b6cfa3e764c77de18cceb5dafb58c0c00bc9b7d6685f7bc59cffe237a3f6338b"} Oct 30 19:00:27.938700 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:27.938385 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerStarted","Data":"ff474ea61d17d7682458638a617ec8d10e81cef318e7f9d071f77e2ff57c69d7"} Oct 30 19:00:31.961535 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:31.961502 6966 generic.go:356] "Generic (PLEG): container finished" podID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerID="b6cfa3e764c77de18cceb5dafb58c0c00bc9b7d6685f7bc59cffe237a3f6338b" exitCode=0 Oct 30 19:00:31.961939 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:31.961546 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerDied","Data":"b6cfa3e764c77de18cceb5dafb58c0c00bc9b7d6685f7bc59cffe237a3f6338b"} Oct 30 19:00:32.940006 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:32.939975 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:32.969191 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:32.969163 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerStarted","Data":"9f82c178dc7d8ce3cca43325e58febf13f55c77d396c7efe310b89affab0b3ed"} Oct 30 19:00:32.969584 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:32.969217 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerStarted","Data":"118db7afa3beec7274ef54399ea10ee360ea51bc9e9f0f9379c05fcf7cc3ae61"} Oct 30 19:00:32.969584 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:32.969241 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerStarted","Data":"60a2f857aabf4cb8d32943b12b0001f03382e02276a35264c335cbe88b16ae14"} Oct 30 19:00:33.038774 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:33.038714 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podStartSLOduration=8.551454276 podStartE2EDuration="12.038697982s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.487243706s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:00:33.036319989 +0000 UTC m=+3795.742505359" watchObservedRunningTime="2026-10-30 19:00:33.038697982 +0000 UTC m=+3795.744883340" Oct 30 19:00:37.095432 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:37.095399 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:42.095263 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:42.095234 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:42.100493 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:42.100472 6966 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:43.022940 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:43.022912 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:47.199873 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:47.199840 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 19:00:50.123996 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.123936 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc"] Oct 30 19:00:50.124704 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.124264 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="kube-rbac-proxy" containerID="cri-o://8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" gracePeriod=30 Oct 30 19:00:50.124704 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.124307 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="prom-label-proxy" containerID="cri-o://674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" gracePeriod=30 Oct 30 19:00:50.124704 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.124362 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="method-gate" containerID="cri-o://ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" gracePeriod=30 Oct 30 19:00:50.145560 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.145534 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c"] Oct 30 19:00:50.145794 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.145759 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" podUID="80238df0-4f32-4963-81ad-82f293141376" containerName="korrel8r" containerID="cri-o://766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480" gracePeriod=30 Oct 30 19:00:50.382139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.382076 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:50.399810 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.399785 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:50.461913 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.461880 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g\" (UniqueName: \"kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g\") pod \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " Oct 30 19:00:50.462087 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.461949 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\" (UniqueName: \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\") pod \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " Oct 30 19:00:50.462259 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.462208 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config\" (UniqueName: \"kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config\") pod \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\" (UID: \"0b834912-5b9f-4a4a-aa48-1e2d48c11bec\") " Oct 30 19:00:50.462507 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.462473 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config" pod "0b834912-5b9f-4a4a-aa48-1e2d48c11bec" (UID: "0b834912-5b9f-4a4a-aa48-1e2d48c11bec"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:50.462732 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.462708 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.464035 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.464014 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g" pod "0b834912-5b9f-4a4a-aa48-1e2d48c11bec" (UID: "0b834912-5b9f-4a4a-aa48-1e2d48c11bec"). InnerVolumeSpecName "kube-api-access-jb69g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:50.464177 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.464162 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls" pod "0b834912-5b9f-4a4a-aa48-1e2d48c11bec" (UID: "0b834912-5b9f-4a4a-aa48-1e2d48c11bec"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:50.563652 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.563617 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\" (UniqueName: \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\") pod \"80238df0-4f32-4963-81ad-82f293141376\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " Oct 30 19:00:50.563843 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.563689 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config\" (UniqueName: \"kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config\") pod \"80238df0-4f32-4963-81ad-82f293141376\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " Oct 30 19:00:50.563843 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.563776 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp\" (UniqueName: \"kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp\") pod \"80238df0-4f32-4963-81ad-82f293141376\" (UID: \"80238df0-4f32-4963-81ad-82f293141376\") " Oct 30 19:00:50.564070 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.564049 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jb69g\" (UniqueName: \"kubernetes.io/projected/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-kube-api-access-jb69g\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564070 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.564070 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/0b834912-5b9f-4a4a-aa48-1e2d48c11bec-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.564273 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.564138 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config" pod "80238df0-4f32-4963-81ad-82f293141376" (UID: "80238df0-4f32-4963-81ad-82f293141376"). InnerVolumeSpecName "config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:50.566217 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.566186 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp" pod "80238df0-4f32-4963-81ad-82f293141376" (UID: "80238df0-4f32-4963-81ad-82f293141376"). InnerVolumeSpecName "kube-api-access-pmhlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:50.566334 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.566284 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls" pod "80238df0-4f32-4963-81ad-82f293141376" (UID: "80238df0-4f32-4963-81ad-82f293141376"). InnerVolumeSpecName "tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:50.603858 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.603827 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 19:00:50.604170 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.604126 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="prometheus" containerID="cri-o://60a2f857aabf4cb8d32943b12b0001f03382e02276a35264c335cbe88b16ae14" gracePeriod=600 Oct 30 19:00:50.604689 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.604657 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="thanos-sidecar" containerID="cri-o://9f82c178dc7d8ce3cca43325e58febf13f55c77d396c7efe310b89affab0b3ed" gracePeriod=600 Oct 30 19:00:50.605506 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.604805 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="config-reloader" containerID="cri-o://118db7afa3beec7274ef54399ea10ee360ea51bc9e9f0f9379c05fcf7cc3ae61" gracePeriod=600 Oct 30 19:00:50.609652 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.609628 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 19:00:50.609951 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.609926 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="alertmanager" containerID="cri-o://2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6" gracePeriod=120 Oct 30 19:00:50.610079 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.609943 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="config-reloader" containerID="cri-o://9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad" gracePeriod=120 Oct 30 19:00:50.628366 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.628338 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:50.664544 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.664485 6966 reconciler_common.go:299] "Volume detached for volume \"tls\" (UniqueName: \"kubernetes.io/secret/80238df0-4f32-4963-81ad-82f293141376-tls\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.664544 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.664506 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/configmap/80238df0-4f32-4963-81ad-82f293141376-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.664544 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.664515 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmhlp\" (UniqueName: \"kubernetes.io/projected/80238df0-4f32-4963-81ad-82f293141376-kube-api-access-pmhlp\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:50.900693 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:50.900657 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:51.061629 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.061602 6966 generic.go:356] "Generic (PLEG): container finished" podID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerID="9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad" exitCode=0 Oct 30 19:00:51.061777 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.061684 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerDied","Data":"9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad"} Oct 30 19:00:51.062998 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.062971 6966 generic.go:356] "Generic (PLEG): container finished" podID="80238df0-4f32-4963-81ad-82f293141376" containerID="766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480" exitCode=0 Oct 30 19:00:51.063122 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.063021 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" Oct 30 19:00:51.063122 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.063048 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" event={"ID":"80238df0-4f32-4963-81ad-82f293141376","Type":"ContainerDied","Data":"766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480"} Oct 30 19:00:51.063122 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.063096 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c" event={"ID":"80238df0-4f32-4963-81ad-82f293141376","Type":"ContainerDied","Data":"374cff1b9120ced4e98333a416620b0af5b596541d56a72d7108f61420dc9c56"} Oct 30 19:00:51.063122 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.063117 6966 scope.go:122] "RemoveContainer" containerID="766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480" Oct 30 19:00:51.065163 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065140 6966 generic.go:356] "Generic (PLEG): container finished" podID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerID="674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" exitCode=0 Oct 30 19:00:51.065163 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065164 6966 generic.go:356] "Generic (PLEG): container finished" podID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerID="ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" exitCode=0 Oct 30 19:00:51.065331 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065177 6966 generic.go:356] "Generic (PLEG): container finished" podID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerID="8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" exitCode=0 Oct 30 19:00:51.065331 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065201 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" Oct 30 19:00:51.065331 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065215 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerDied","Data":"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6"} Oct 30 19:00:51.065331 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065283 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerDied","Data":"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046"} Oct 30 19:00:51.065331 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065325 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerDied","Data":"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392"} Oct 30 19:00:51.065666 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.065363 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc" event={"ID":"0b834912-5b9f-4a4a-aa48-1e2d48c11bec","Type":"ContainerDied","Data":"96373640c8d522ec84df1c5660b2051eaf2b507e082e37c53b9157e3bd23f0ae"} Oct 30 19:00:51.068389 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.068364 6966 generic.go:356] "Generic (PLEG): container finished" podID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerID="9f82c178dc7d8ce3cca43325e58febf13f55c77d396c7efe310b89affab0b3ed" exitCode=0 Oct 30 19:00:51.068505 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.068391 6966 generic.go:356] "Generic (PLEG): container finished" podID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerID="118db7afa3beec7274ef54399ea10ee360ea51bc9e9f0f9379c05fcf7cc3ae61" exitCode=0 Oct 30 19:00:51.068505 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.068401 6966 generic.go:356] "Generic (PLEG): container finished" podID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerID="60a2f857aabf4cb8d32943b12b0001f03382e02276a35264c335cbe88b16ae14" exitCode=0 Oct 30 19:00:51.068601 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.068542 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerDied","Data":"9f82c178dc7d8ce3cca43325e58febf13f55c77d396c7efe310b89affab0b3ed"} Oct 30 19:00:51.068601 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.068596 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerDied","Data":"118db7afa3beec7274ef54399ea10ee360ea51bc9e9f0f9379c05fcf7cc3ae61"} Oct 30 19:00:51.068687 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.068633 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerDied","Data":"60a2f857aabf4cb8d32943b12b0001f03382e02276a35264c335cbe88b16ae14"} Oct 30 19:00:51.073156 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.073141 6966 scope.go:122] "RemoveContainer" containerID="766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480" Oct 30 19:00:51.073398 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:51.073379 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480\": container with ID starting with 766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480 not found: ID does not exist" containerID="766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480" Oct 30 19:00:51.073501 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.073409 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480"} err="failed to get container status \"766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480\": rpc error: code = NotFound desc = could not find container \"766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480\": container with ID starting with 766170a9ca6c8161e352769b7f0fc22a3bae890ffeb6219699d7614126f4a480 not found: ID does not exist" Oct 30 19:00:51.073501 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.073430 6966 scope.go:122] "RemoveContainer" containerID="674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" Oct 30 19:00:51.081809 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.081793 6966 scope.go:122] "RemoveContainer" containerID="ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" Oct 30 19:00:51.088957 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.088943 6966 scope.go:122] "RemoveContainer" containerID="8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" Oct 30 19:00:51.096220 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.096204 6966 scope.go:122] "RemoveContainer" containerID="674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" Oct 30 19:00:51.096486 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:51.096450 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": container with ID starting with 674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6 not found: ID does not exist" containerID="674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" Oct 30 19:00:51.096575 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.096495 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6"} err="failed to get container status \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": rpc error: code = NotFound desc = could not find container \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": container with ID starting with 674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6 not found: ID does not exist" Oct 30 19:00:51.096575 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.096517 6966 scope.go:122] "RemoveContainer" containerID="ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" Oct 30 19:00:51.096757 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:51.096739 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": container with ID starting with ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046 not found: ID does not exist" containerID="ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" Oct 30 19:00:51.096801 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.096763 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046"} err="failed to get container status \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": rpc error: code = NotFound desc = could not find container \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": container with ID starting with ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046 not found: ID does not exist" Oct 30 19:00:51.096801 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.096778 6966 scope.go:122] "RemoveContainer" containerID="8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" Oct 30 19:00:51.097018 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:51.097004 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": container with ID starting with 8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392 not found: ID does not exist" containerID="8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" Oct 30 19:00:51.097064 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097021 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392"} err="failed to get container status \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": rpc error: code = NotFound desc = could not find container \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": container with ID starting with 8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392 not found: ID does not exist" Oct 30 19:00:51.097064 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097032 6966 scope.go:122] "RemoveContainer" containerID="674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" Oct 30 19:00:51.097266 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097247 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6"} err="failed to get container status \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": rpc error: code = NotFound desc = could not find container \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": container with ID starting with 674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6 not found: ID does not exist" Oct 30 19:00:51.097329 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097266 6966 scope.go:122] "RemoveContainer" containerID="ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" Oct 30 19:00:51.097538 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097515 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046"} err="failed to get container status \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": rpc error: code = NotFound desc = could not find container \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": container with ID starting with ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046 not found: ID does not exist" Oct 30 19:00:51.097627 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097539 6966 scope.go:122] "RemoveContainer" containerID="8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" Oct 30 19:00:51.097816 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097793 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392"} err="failed to get container status \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": rpc error: code = NotFound desc = could not find container \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": container with ID starting with 8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392 not found: ID does not exist" Oct 30 19:00:51.097895 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.097816 6966 scope.go:122] "RemoveContainer" containerID="674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6" Oct 30 19:00:51.098063 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.098043 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6"} err="failed to get container status \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": rpc error: code = NotFound desc = could not find container \"674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6\": container with ID starting with 674826e8bd68d2d9d4c750196ac1488592ba938c7f0ca948f4cd96e7ddf600a6 not found: ID does not exist" Oct 30 19:00:51.098141 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.098064 6966 scope.go:122] "RemoveContainer" containerID="ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046" Oct 30 19:00:51.098288 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.098269 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046"} err="failed to get container status \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": rpc error: code = NotFound desc = could not find container \"ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046\": container with ID starting with ed2396d6294661c86315a8c58305856a164d019301d21bc25d73ea77356de046 not found: ID does not exist" Oct 30 19:00:51.098359 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.098290 6966 scope.go:122] "RemoveContainer" containerID="8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392" Oct 30 19:00:51.098530 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.098513 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392"} err="failed to get container status \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": rpc error: code = NotFound desc = could not find container \"8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392\": container with ID starting with 8b1731754c695aa4dca12f403f88007d5845a9261c98c38cc6ec1c1917891392 not found: ID does not exist" Oct 30 19:00:51.112413 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.112391 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc"] Oct 30 19:00:51.115627 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.115608 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-thanos-querier-proxy-6998ff8b98-hh6qc"] Oct 30 19:00:51.134554 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.134521 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:51.147963 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.147930 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c"] Oct 30 19:00:51.152381 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.152361 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/korrel8r-5cbcc89b8f-lgv8c"] Oct 30 19:00:51.319372 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.300112 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:51.319372 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.300422 6966 kuberuntime_container.go:898] "Killing container with a grace period" pod="redhat-ods-monitoring/data-science-collector-collector-1" podUID="b40bf11b-907a-451f-bd41-14cf7ec85544" containerName="otc-container" containerID="cri-o://2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf" gracePeriod=30 Oct 30 19:00:51.414884 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.414013 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" path="/var/lib/kubelet/pods/0b834912-5b9f-4a4a-aa48-1e2d48c11bec/volumes" Oct 30 19:00:51.414884 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.414649 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80238df0-4f32-4963-81ad-82f293141376" path="/var/lib/kubelet/pods/80238df0-4f32-4963-81ad-82f293141376/volumes" Oct 30 19:00:51.443637 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.443615 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:51.463183 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.463156 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:51.489574 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.489512 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:51.570731 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570700 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.570895 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570756 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.570895 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570791 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.570895 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570833 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.570895 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570868 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.571139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570913 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.571139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570944 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.571139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.570980 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571020 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571052 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "alertmanager-data-science-monitoringstack-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:51.571139 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571065 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571153 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571185 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571278 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571329 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571363 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571402 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571504 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571476 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571873 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571515 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config\") pod \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\" (UID: \"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35\") " Oct 30 19:00:51.571873 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571540 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.571873 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571573 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0\") pod \"6cfe905f-2910-478c-9712-68ba0b12ae3f\" (UID: \"6cfe905f-2910-478c-9712-68ba0b12ae3f\") " Oct 30 19:00:51.572053 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.571908 6966 reconciler_common.go:299] "Volume detached for volume \"alertmanager-data-science-monitoringstack-db\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-alertmanager-data-science-monitoringstack-db\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.572424 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.572394 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.573726 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.573697 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.574664 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.574620 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "prometheus-data-science-monitoringstack-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.575127 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.575106 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:51.575988 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.575941 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "kube-api-access-5zm99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.576080 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.575995 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.577432 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577311 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.577432 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577363 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "web-config-tls-secret-cert-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.577432 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577386 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "kube-api-access-7qvnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.577432 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577425 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "web-config-tls-secret-key-prometheus-operated-tls-f3457c70". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.577738 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577593 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.577829 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577786 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:51.577967 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.577845 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.579863 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.579836 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 19:00:51.580427 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.580403 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "secret-prometheus-web-tls-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.580909 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.580884 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7". PluginName "kubernetes.io/csi", VolumeGIDValue "" Oct 30 19:00:51.582329 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.582309 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.584144 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.584119 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.589629 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.589601 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config" pod "6cfe905f-2910-478c-9712-68ba0b12ae3f" (UID: "6cfe905f-2910-478c-9712-68ba0b12ae3f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.591447 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.591427 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config" pod "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" (UID: "6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.673162 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673129 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs\") pod \"b40bf11b-907a-451f-bd41-14cf7ec85544\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " Oct 30 19:00:51.673162 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673167 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal\" (UniqueName: \"kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal\") pod \"b40bf11b-907a-451f-bd41-14cf7ec85544\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " Oct 30 19:00:51.673382 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673189 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs\") pod \"b40bf11b-907a-451f-bd41-14cf7ec85544\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " Oct 30 19:00:51.673382 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673220 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert\") pod \"b40bf11b-907a-451f-bd41-14cf7ec85544\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " Oct 30 19:00:51.673382 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673310 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v\" (UniqueName: \"kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v\") pod \"b40bf11b-907a-451f-bd41-14cf7ec85544\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " Oct 30 19:00:51.673382 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673362 6966 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs\") pod \"b40bf11b-907a-451f-bd41-14cf7ec85544\" (UID: \"b40bf11b-907a-451f-bd41-14cf7ec85544\") " Oct 30 19:00:51.673712 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673613 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal" pod "b40bf11b-907a-451f-bd41-14cf7ec85544" (UID: "b40bf11b-907a-451f-bd41-14cf7ec85544"). InnerVolumeSpecName "otc-internal". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 19:00:51.673712 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673631 6966 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673712 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673662 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-1\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673712 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673697 6966 reconciler_common.go:292] "operationExecutor.UnmountDevice started for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" " Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673716 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-2\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673734 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673750 6966 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-web-tls-ca\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-secret-prometheus-web-tls-ca\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673762 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6cfe905f-2910-478c-9712-68ba0b12ae3f-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673775 6966 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-cluster-tls-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673788 6966 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673800 6966 reconciler_common.go:299] "Volume detached for volume \"prometheus-data-science-monitoringstack-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6cfe905f-2910-478c-9712-68ba0b12ae3f-prometheus-data-science-monitoringstack-rulefiles-0\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673815 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-key-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-key-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673831 6966 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673845 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qvnc\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-kube-api-access-7qvnc\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673860 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673872 6966 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-out\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673883 6966 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35-config-volume\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673896 6966 reconciler_common.go:299] "Volume detached for volume \"web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\" (UniqueName: \"kubernetes.io/secret/6cfe905f-2910-478c-9712-68ba0b12ae3f-web-config-tls-secret-cert-prometheus-operated-tls-f3457c70\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673909 6966 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-tls-assets\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.673960 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.673922 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zm99\" (UniqueName: \"kubernetes.io/projected/6cfe905f-2910-478c-9712-68ba0b12ae3f-kube-api-access-5zm99\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.675850 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.675825 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs" pod "b40bf11b-907a-451f-bd41-14cf7ec85544" (UID: "b40bf11b-907a-451f-bd41-14cf7ec85544"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.675974 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.675855 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v" pod "b40bf11b-907a-451f-bd41-14cf7ec85544" (UID: "b40bf11b-907a-451f-bd41-14cf7ec85544"). InnerVolumeSpecName "kube-api-access-kbx8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 19:00:51.676041 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.676012 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs" pod "b40bf11b-907a-451f-bd41-14cf7ec85544" (UID: "b40bf11b-907a-451f-bd41-14cf7ec85544"). InnerVolumeSpecName "otlp-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.676093 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.676059 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert" pod "b40bf11b-907a-451f-bd41-14cf7ec85544" (UID: "b40bf11b-907a-451f-bd41-14cf7ec85544"). InnerVolumeSpecName "data-science-collector-ta-client-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.676281 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.676264 6966 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs" pod "b40bf11b-907a-451f-bd41-14cf7ec85544" (UID: "b40bf11b-907a-451f-bd41-14cf7ec85544"). InnerVolumeSpecName "monitoring-tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 19:00:51.685646 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.685631 6966 operation_generator.go:893] UnmountDevice succeeded for volume "pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7" (UniqueName: "kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a") on node "ip-10-0-136-241.ec2.internal" Oct 30 19:00:51.695879 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.695851 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:51.774837 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774785 6966 reconciler_common.go:299] "Volume detached for volume \"pvc-bd619049-8a54-4160-91d7-5ec48e0d63a7\" (UniqueName: \"kubernetes.io/csi/ebs.csi.aws.com^vol-06bab37386b4abf2a\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"csi-016ea238436eac68797204531fb11898c471725e0c038b7818d13acf333c5790\"" Oct 30 19:00:51.774837 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774810 6966 reconciler_common.go:299] "Volume detached for volume \"otlp-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-otlp-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.774837 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774820 6966 reconciler_common.go:299] "Volume detached for volume \"monitoring-tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-monitoring-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.774837 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774829 6966 reconciler_common.go:299] "Volume detached for volume \"otc-internal\" (UniqueName: \"kubernetes.io/configmap/b40bf11b-907a-451f-bd41-14cf7ec85544-otc-internal\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.774837 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774838 6966 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-tls-certs\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.775064 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774847 6966 reconciler_common.go:299] "Volume detached for volume \"data-science-collector-ta-client-cert\" (UniqueName: \"kubernetes.io/secret/b40bf11b-907a-451f-bd41-14cf7ec85544-data-science-collector-ta-client-cert\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:51.775064 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:51.774856 6966 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbx8v\" (UniqueName: \"kubernetes.io/projected/b40bf11b-907a-451f-bd41-14cf7ec85544-kube-api-access-kbx8v\") on node \"ip-10-0-136-241.ec2.internal\" DevicePath \"\"" Oct 30 19:00:52.080052 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.080018 6966 generic.go:356] "Generic (PLEG): container finished" podID="b40bf11b-907a-451f-bd41-14cf7ec85544" containerID="2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf" exitCode=0 Oct 30 19:00:52.080231 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.080100 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/data-science-collector-collector-1" Oct 30 19:00:52.080231 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.080107 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b40bf11b-907a-451f-bd41-14cf7ec85544","Type":"ContainerDied","Data":"2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf"} Oct 30 19:00:52.080231 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.080201 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/data-science-collector-collector-1" event={"ID":"b40bf11b-907a-451f-bd41-14cf7ec85544","Type":"ContainerDied","Data":"024f7f1dfbaa7534de93218aed0736d722457f60744c15fc0ebaec36a1a5de2c"} Oct 30 19:00:52.080231 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.080229 6966 scope.go:122] "RemoveContainer" containerID="2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf" Oct 30 19:00:52.083711 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.082956 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" Oct 30 19:00:52.083711 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.082955 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/prometheus-data-science-monitoringstack-0" event={"ID":"6cfe905f-2910-478c-9712-68ba0b12ae3f","Type":"ContainerDied","Data":"ff474ea61d17d7682458638a617ec8d10e81cef318e7f9d071f77e2ff57c69d7"} Oct 30 19:00:52.085473 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.085439 6966 generic.go:356] "Generic (PLEG): container finished" podID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerID="2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6" exitCode=0 Oct 30 19:00:52.085585 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.085542 6966 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" Oct 30 19:00:52.085670 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.085534 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerDied","Data":"2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6"} Oct 30 19:00:52.085734 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.085684 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0" event={"ID":"6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35","Type":"ContainerDied","Data":"55f175f00b33c5bb1020bbd2c35150b1e98b11024b6cfb2d300e98805f707793"} Oct 30 19:00:52.095060 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.095043 6966 scope.go:122] "RemoveContainer" containerID="2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf" Oct 30 19:00:52.095319 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:52.095300 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf\": container with ID starting with 2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf not found: ID does not exist" containerID="2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf" Oct 30 19:00:52.095384 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.095327 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf"} err="failed to get container status \"2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf\": rpc error: code = NotFound desc = could not find container \"2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf\": container with ID starting with 2005bc005c4123b1daa637cc76c1f4f22f5110e5265f1d80702710caa612c6cf not found: ID does not exist" Oct 30 19:00:52.095384 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.095342 6966 scope.go:122] "RemoveContainer" containerID="9f82c178dc7d8ce3cca43325e58febf13f55c77d396c7efe310b89affab0b3ed" Oct 30 19:00:52.103472 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.103439 6966 scope.go:122] "RemoveContainer" containerID="118db7afa3beec7274ef54399ea10ee360ea51bc9e9f0f9379c05fcf7cc3ae61" Oct 30 19:00:52.112617 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.112599 6966 scope.go:122] "RemoveContainer" containerID="60a2f857aabf4cb8d32943b12b0001f03382e02276a35264c335cbe88b16ae14" Oct 30 19:00:52.120503 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.120484 6966 scope.go:122] "RemoveContainer" containerID="b6cfa3e764c77de18cceb5dafb58c0c00bc9b7d6685f7bc59cffe237a3f6338b" Oct 30 19:00:52.128274 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.128260 6966 scope.go:122] "RemoveContainer" containerID="9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad" Oct 30 19:00:52.159808 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.159785 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:52.161111 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.161094 6966 scope.go:122] "RemoveContainer" containerID="2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6" Oct 30 19:00:52.162353 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.162332 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/data-science-collector-collector-1"] Oct 30 19:00:52.169024 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.169009 6966 scope.go:122] "RemoveContainer" containerID="039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1" Oct 30 19:00:52.179009 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.178987 6966 scope.go:122] "RemoveContainer" containerID="9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad" Oct 30 19:00:52.179289 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:52.179266 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad\": container with ID starting with 9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad not found: ID does not exist" containerID="9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad" Oct 30 19:00:52.179356 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.179298 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad"} err="failed to get container status \"9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad\": rpc error: code = NotFound desc = could not find container \"9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad\": container with ID starting with 9c5f3ad4382a1c034143357030370d984f7536fed9792c8aaf45d70fbed192ad not found: ID does not exist" Oct 30 19:00:52.179356 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.179319 6966 scope.go:122] "RemoveContainer" containerID="2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6" Oct 30 19:00:52.179654 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:52.179628 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6\": container with ID starting with 2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6 not found: ID does not exist" containerID="2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6" Oct 30 19:00:52.179713 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.179665 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6"} err="failed to get container status \"2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6\": rpc error: code = NotFound desc = could not find container \"2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6\": container with ID starting with 2a70bad8251eaf054f05d6e036b1b2bc4ec39b3813b00af89ad2d6f8ec6c95d6 not found: ID does not exist" Oct 30 19:00:52.179713 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.179689 6966 scope.go:122] "RemoveContainer" containerID="039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1" Oct 30 19:00:52.180015 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:00:52.179992 6966 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1\": container with ID starting with 039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1 not found: ID does not exist" containerID="039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1" Oct 30 19:00:52.180129 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.180016 6966 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1"} err="failed to get container status \"039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1\": rpc error: code = NotFound desc = could not find container \"039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1\": container with ID starting with 039f82e7fd2e3b35f0f863b2e119cf10b2f27a607a9cc150b17629ec978bbca1 not found: ID does not exist" Oct 30 19:00:52.199697 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.199484 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 19:00:52.205081 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.205057 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/alertmanager-data-science-monitoringstack-0"] Oct 30 19:00:52.242203 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.242178 6966 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 19:00:52.248768 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.248744 6966 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["redhat-ods-monitoring/prometheus-data-science-monitoringstack-0"] Oct 30 19:00:52.350906 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.350821 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:52.572831 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:52.572789 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:53.410328 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:53.410294 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" path="/var/lib/kubelet/pods/6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35/volumes" Oct 30 19:00:53.410829 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:53.410814 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" path="/var/lib/kubelet/pods/6cfe905f-2910-478c-9712-68ba0b12ae3f/volumes" Oct 30 19:00:53.411574 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:53.411558 6966 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b40bf11b-907a-451f-bd41-14cf7ec85544" path="/var/lib/kubelet/pods/b40bf11b-907a-451f-bd41-14cf7ec85544/volumes" Oct 30 19:00:54.388812 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:54.388781 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:00:54.611746 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:00:54.611713 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:08.735962 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:08.735927 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:14.453053 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:14.453015 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:14.723490 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:14.723396 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:17.200127 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:01:17.200085 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 19:01:18.523076 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:18.523044 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-observability-system_odh-observability-5999d9c564-mm8f7_eccd2c86-b51e-482f-9116-ddecf6a1b930/manager/2.log" Oct 30 19:01:19.095052 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:19.095020 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:19.333422 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:19.333387 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:23.525452 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:23.525352 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gqtbs_788e0688-d9fb-48af-993d-c3bc6a0bc4c5/global-pull-secret-syncer/0.log" Oct 30 19:01:24.415559 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:24.415520 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" Oct 30 19:01:24.473373 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:24.473345 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:25.098859 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:25.098829 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-241.ec2.internal_ef2a11081781848cec6e4c7eceee10f4/haproxy/0.log" Oct 30 19:01:25.483750 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:25.483666 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:28.023695 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:28.023655 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:30.138707 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.138678 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/alertmanager/0.log" Oct 30 19:01:30.269502 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.269474 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/config-reloader/0.log" Oct 30 19:01:30.399138 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.399068 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/kube-rbac-proxy-web/0.log" Oct 30 19:01:30.526414 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.526395 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/kube-rbac-proxy/0.log" Oct 30 19:01:30.658127 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.658061 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/kube-rbac-proxy-metric/0.log" Oct 30 19:01:30.699409 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.699383 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:30.786167 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.786147 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/prom-label-proxy/0.log" Oct 30 19:01:30.918253 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.918186 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_b5b9edbf-0560-47ba-8c1b-aecca3a6365a/init-config-reloader/0.log" Oct 30 19:01:30.925477 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:30.925430 6966 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-bzkm6_cfcad2e0-540b-4e42-ba77-303e060d37bd/konnectivity-agent/0.log" limit=1024 Oct 30 19:01:31.197547 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.197486 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dm4fv_eb1042dd-c449-4c9e-9a56-899e68938400/kube-state-metrics/0.log" Oct 30 19:01:31.318956 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.318925 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dm4fv_eb1042dd-c449-4c9e-9a56-899e68938400/kube-rbac-proxy-main/0.log" Oct 30 19:01:31.447378 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.447350 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-dm4fv_eb1042dd-c449-4c9e-9a56-899e68938400/kube-rbac-proxy-self/0.log" Oct 30 19:01:31.579483 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.579438 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5c6d98fbbf-ckgjf_cd87de0b-e92c-4740-ab77-b5793a227d40/metrics-server/0.log" Oct 30 19:01:31.709476 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.709438 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-bmk6p_447f1a32-218e-4599-9a99-011281767a95/monitoring-plugin/0.log" Oct 30 19:01:31.852683 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.852618 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fgr8_92f93306-6430-45d2-887a-f056426c57ed/node-exporter/0.log" Oct 30 19:01:31.977988 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:31.977939 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fgr8_92f93306-6430-45d2-887a-f056426c57ed/kube-rbac-proxy/0.log" Oct 30 19:01:32.105260 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.105208 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4fgr8_92f93306-6430-45d2-887a-f056426c57ed/init-textfile/0.log" Oct 30 19:01:32.708568 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.708537 6966 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j"] Oct 30 19:01:32.709109 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709086 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="prom-label-proxy" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709112 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="prom-label-proxy" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709128 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="thanos-sidecar" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709135 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="thanos-sidecar" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709153 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="alertmanager" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709161 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="alertmanager" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709179 6966 cpu_manager.go:414] "containerMap: removing container" podUID="b40bf11b-907a-451f-bd41-14cf7ec85544" containerName="otc-container" Oct 30 19:01:32.709196 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709187 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b40bf11b-907a-451f-bd41-14cf7ec85544" containerName="otc-container" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709204 6966 cpu_manager.go:414] "containerMap: removing container" podUID="80238df0-4f32-4963-81ad-82f293141376" containerName="korrel8r" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709212 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80238df0-4f32-4963-81ad-82f293141376" containerName="korrel8r" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709224 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="init-config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709231 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="init-config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709241 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="prometheus" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709248 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="prometheus" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709263 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="kube-rbac-proxy" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709270 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="kube-rbac-proxy" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709284 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709291 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709300 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709307 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709332 6966 cpu_manager.go:414] "containerMap: removing container" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="method-gate" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709339 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="method-gate" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709355 6966 cpu_manager.go:414] "containerMap: removing container" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="init-config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709362 6966 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="init-config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709437 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="prom-label-proxy" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709450 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="thanos-sidecar" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709474 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="alertmanager" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709485 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="b40bf11b-907a-451f-bd41-14cf7ec85544" containerName="otc-container" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709495 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="80238df0-4f32-4963-81ad-82f293141376" containerName="korrel8r" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709502 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="prometheus" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709511 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="kube-rbac-proxy" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709520 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6695d7ff-9c2a-45e2-bbc2-24e4bd00fc35" containerName="config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709529 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cfe905f-2910-478c-9712-68ba0b12ae3f" containerName="config-reloader" Oct 30 19:01:32.709600 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.709540 6966 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b834912-5b9f-4a4a-aa48-1e2d48c11bec" containerName="method-gate" Oct 30 19:01:32.712984 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.712954 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.730584 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.730566 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ht9cb\"/\"openshift-service-ca.crt\"" Oct 30 19:01:32.730679 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.730593 6966 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ht9cb\"/\"kube-root-ca.crt\"" Oct 30 19:01:32.738701 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.738683 6966 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ht9cb\"/\"default-dockercfg-b858q\"" Oct 30 19:01:32.740316 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.740299 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j"] Oct 30 19:01:32.781193 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.781165 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-lib-modules\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.781347 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.781211 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-sys\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.781425 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.781345 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnkc9\" (UniqueName: \"kubernetes.io/projected/bc4aa579-cd12-41bc-8d32-cfcadfa14150-kube-api-access-hnkc9\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.781425 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.781399 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-proc\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.781542 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.781487 6966 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-podres\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882559 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882531 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnkc9\" (UniqueName: \"kubernetes.io/projected/bc4aa579-cd12-41bc-8d32-cfcadfa14150-kube-api-access-hnkc9\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882678 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882569 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-proc\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882678 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882586 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-podres\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882678 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882638 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-proc\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882678 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882663 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-lib-modules\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882845 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882691 6966 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-sys\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882845 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882716 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-podres\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882845 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882746 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-sys\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.882845 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.882780 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc4aa579-cd12-41bc-8d32-cfcadfa14150-lib-modules\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:32.905782 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:32.905757 6966 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnkc9\" (UniqueName: \"kubernetes.io/projected/bc4aa579-cd12-41bc-8d32-cfcadfa14150-kube-api-access-hnkc9\") pod \"perf-node-gather-daemonset-mtp6j\" (UID: \"bc4aa579-cd12-41bc-8d32-cfcadfa14150\") " pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:33.019357 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:33.019295 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fqklq_185b41a3-2575-4865-ba56-77b1c1b40989/kube-rbac-proxy-main/0.log" Oct 30 19:01:33.025247 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:33.025228 6966 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:33.146854 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:33.146821 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fqklq_185b41a3-2575-4865-ba56-77b1c1b40989/kube-rbac-proxy-self/0.log" Oct 30 19:01:33.280704 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:33.280616 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-fqklq_185b41a3-2575-4865-ba56-77b1c1b40989/openshift-state-metrics/0.log" Oct 30 19:01:33.319940 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:33.319911 6966 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j"] Oct 30 19:01:33.325128 ip-10-0-136-241 kubenswrapper[6966]: W1030 19:01:33.325102 6966 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbc4aa579_cd12_41bc_8d32_cfcadfa14150.slice/crio-1e7fc6c57d2870a50ffc082464fa3c92057960077322387195f954e4e581380c WatchSource:0}: Error finding container 1e7fc6c57d2870a50ffc082464fa3c92057960077322387195f954e4e581380c: Status 404 returned error can't find the container with id 1e7fc6c57d2870a50ffc082464fa3c92057960077322387195f954e4e581380c Oct 30 19:01:34.308761 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:34.308729 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" event={"ID":"bc4aa579-cd12-41bc-8d32-cfcadfa14150","Type":"ContainerStarted","Data":"c8d32ab626981ce27cdd1689c6e1416f58cdc90790bc93c42cd101206484d76a"} Oct 30 19:01:34.309184 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:34.308778 6966 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" event={"ID":"bc4aa579-cd12-41bc-8d32-cfcadfa14150","Type":"ContainerStarted","Data":"1e7fc6c57d2870a50ffc082464fa3c92057960077322387195f954e4e581380c"} Oct 30 19:01:34.309184 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:34.308812 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:34.350627 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:34.350584 6966 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" podStartSLOduration=2.350570443 podStartE2EDuration="2.350570443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 19:01:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 19:01:34.349581189 +0000 UTC m=+3857.055766525" watchObservedRunningTime="2026-10-30 19:01:34.350570443 +0000 UTC m=+3857.056755800" Oct 30 19:01:35.152023 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:35.151993 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bc75cc55-5scfq_d35b6637-df31-4561-9b9e-ebf305bd997f/thanos-query/0.log" Oct 30 19:01:35.279606 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:35.279579 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bc75cc55-5scfq_d35b6637-df31-4561-9b9e-ebf305bd997f/kube-rbac-proxy-web/0.log" Oct 30 19:01:35.407555 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:35.407492 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bc75cc55-5scfq_d35b6637-df31-4561-9b9e-ebf305bd997f/kube-rbac-proxy/0.log" Oct 30 19:01:35.535844 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:35.535819 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bc75cc55-5scfq_d35b6637-df31-4561-9b9e-ebf305bd997f/prom-label-proxy/0.log" Oct 30 19:01:35.662917 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:35.662850 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bc75cc55-5scfq_d35b6637-df31-4561-9b9e-ebf305bd997f/kube-rbac-proxy-rules/0.log" Oct 30 19:01:35.790314 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:35.790290 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-67bc75cc55-5scfq_d35b6637-df31-4561-9b9e-ebf305bd997f/kube-rbac-proxy-metrics/0.log" Oct 30 19:01:39.078224 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:39.078191 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-58cbf4966f-qjsxr_b8c7525d-92a7-400b-8367-b88234b4a79c/console/0.log" Oct 30 19:01:40.322219 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:40.322193 6966 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ht9cb/perf-node-gather-daemonset-mtp6j" Oct 30 19:01:43.002479 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:43.002427 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5c26_699b6cb3-446f-440e-8d68-66c4e09ee322/dns/0.log" Oct 30 19:01:43.132881 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:43.132850 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-z5c26_699b6cb3-446f-440e-8d68-66c4e09ee322/kube-rbac-proxy/0.log" Oct 30 19:01:43.383000 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:43.382972 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-l2ddq_bbd7a723-acf0-4c7c-95a5-2bd54d65460d/dns-node-resolver/0.log" Oct 30 19:01:44.868790 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:44.868760 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rd5c5_9cc87edf-5705-49bc-abe7-7cf921ed75ab/node-ca/0.log" Oct 30 19:01:47.198852 ip-10-0-136-241 kubenswrapper[6966]: E1030 19:01:47.198819 6966 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-56qxs" containerName="extractor" Oct 30 19:01:47.976631 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:47.976591 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vm92t_bc0cafd7-e383-4391-8d85-073289c3f442/serve-healthcheck-canary/0.log" Oct 30 19:01:49.041084 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:49.041054 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56qxs_85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/kube-rbac-proxy/0.log" Oct 30 19:01:49.172944 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:49.172913 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56qxs_85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/exporter/0.log" Oct 30 19:01:49.308433 ip-10-0-136-241 kubenswrapper[6966]: I1030 19:01:49.308367 6966 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-56qxs_85b2e7b1-1141-4cb4-a0fa-5376cf8014d8/extractor/0.log"